Bag of Words (BoW)

Bag of Words (BoW) represents a document as a vector of word counts, completely ignoring word order and grammar — the document is treated as an unordered "bag" of its words.

Building a BoW Model — Step by Step

documents = [
    "I love NLP",
    "I love Python",
    "NLP and Python are fun"
]

# Step 1: Build the vocabulary (all unique words across all documents)
vocabulary = sorted(set(" ".join(documents).lower().split()))
print(vocabulary)
# ['and', 'are', 'fun', 'i', 'love', 'nlp', 'python']

# Step 2: For each document, count occurrences of each vocabulary word
def bow_vector(doc, vocab):
    words = doc.lower().split()
    return [words.count(v) for v in vocab]

for doc in documents:
    print(doc, "->", bow_vector(doc, vocabulary))
# I love NLP                 -> [0, 0, 0, 1, 1, 1, 0]
# I love Python               -> [0, 0, 0, 1, 1, 0, 1]
# NLP and Python are fun       -> [1, 1, 1, 0, 0, 1, 1]

Using scikit-learn's CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer

documents = ["I love NLP", "I love Python", "NLP and Python are fun"]

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(documents)

print(vectorizer.get_feature_names_out())
# ['and' 'are' 'fun' 'love' 'nlp' 'python']    (note: single-letter "I" dropped by default token pattern)
print(bow_matrix.toarray())
# [[0 0 0 1 1 0]
#  [0 0 0 1 0 1]
#  [1 1 1 0 1 1]]

BoW with N-grams (Capturing Some Word Order)

Plain BoW loses all order ("dog bites man" and "man bites dog" get identical unigram vectors!). Using n-grams as vocabulary units recovers some local order:

vectorizer = CountVectorizer(ngram_range=(1, 2))  # unigrams AND bigrams
docs = ["dog bites man", "man bites dog"]
matrix = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
# ['bites' 'bites dog' 'bites man' 'dog' 'dog bites' 'man' 'man bites']
print(matrix.toarray())
# [[1 0 1 1 1 1 0]
#  [1 1 0 1 0 1 1]]
# Now the two sentences produce DIFFERENT vectors!

Limiting Vocabulary Size

vectorizer = CountVectorizer(max_features=1000, min_df=2, max_df=0.9)
# max_features: keep only the 1000 most frequent terms
# min_df=2:     ignore terms appearing in fewer than 2 documents (rare/noisy words)
# max_df=0.9:   ignore terms appearing in more than 90% of documents (too common, low signal)

Limitations of Bag of Words

The third limitation — treating all words as equally important — is directly addressed by TF-IDF, covered next. The last limitation (no semantic similarity) is solved by word embeddings in Unit 4.