TF-IDF (Term Frequency–Inverse Document Frequency)

Plain Bag-of-Words treats every word's raw count as equally meaningful — but common words like "the" appear frequently everywhere and carry little discriminative signal, while rare, topic-specific words are far more informative. TF-IDF fixes this by weighting terms by both how often they appear in a document AND how rare they are across the corpus.

The Formula

TF-IDF(t, d) = TF(t, d) × IDF(t)

TF(t, d)  = (Number of times term t appears in document d) / (Total terms in d)

IDF(t)    = log( N / (1 + df(t)) )
            where N = total number of documents
                  df(t) = number of documents containing term t

• TF — how important the word is within this one document.
• IDF — how rare/distinctive the word is across the whole corpus (common words get a low IDF, near zero; rare words get a high IDF).

Worked Example by Hand

import math

documents = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are great pets"
]

def tf(term, doc):
    words = doc.split()
    return words.count(term) / len(words)

def idf(term, docs):
    N = len(docs)
    df = sum(1 for d in docs if term in d.split())
    return math.log(N / (1 + df))

term = "the"
print("TF:", tf(term, documents[0]))    # 2/6 = 0.333
print("IDF:", idf(term, documents))     # log(3/3) = 0.0  -- appears in 2 of 3 docs, low IDF
print("TF-IDF:", tf(term, documents[0]) * idf(term, documents))  # ~0.0

term2 = "cat"
print("TF:", tf(term2, documents[0]))   # 1/6 = 0.167
print("IDF:", idf(term2, documents))    # log(3/2) = 0.405 -- rare, higher IDF
print("TF-IDF:", tf(term2, documents[0]) * idf(term2, documents))  # ~0.068

"the" gets a near-zero TF-IDF score (it's everywhere, so it's uninformative), while "cat" — which only appears in one document — gets a meaningfully higher score.

Using scikit-learn's TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are great pets"
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(documents)

print(vectorizer.get_feature_names_out())
import pandas as pd
df = pd.DataFrame(tfidf_matrix.toarray(), columns=vectorizer.get_feature_names_out())
print(df.round(3))
#     and   are   cat  cats   dog  dogs  great   log   mat    on   pets   sat  the
# 0  0.00  0.00  0.42  0.00  0.00  0.00   0.00  0.00  0.42  0.32  0.00  0.32  0.65
# 1  0.00  0.00  0.00  0.00  0.42  0.00   0.00  0.42  0.00  0.32  0.00  0.32  0.65
# 2  0.41  0.41  0.00  0.41  0.00  0.41   0.41  0.00  0.00  0.00  0.41  0.00  0.00

Note "the" — appearing in 2 of 3 documents — gets weighted down relative to distinctive words like "mat" or "pets".

BoW vs TF-IDF

TF-IDF for Document Similarity (Cosine Similarity)

from sklearn.metrics.pairwise import cosine_similarity

similarity = cosine_similarity(tfidf_matrix[0], tfidf_matrix[1])
print(similarity)
# [[0.409]]   -- doc0 ("cat...mat") and doc1 ("dog...log") share structure, moderate similarity

similarity2 = cosine_similarity(tfidf_matrix[0], tfidf_matrix[2])
print(similarity2)
# [[0.0]]     -- doc0 and doc2 share almost no vocabulary, near-zero similarity

TF-IDF vectors are the standard input to the Naïve Bayes text classifier, covered in the next lesson.