Text Preprocessing — Stop-word Removal

Stop words are extremely common words (articles, prepositions, pronouns, conjunctions) that carry little topical/semantic meaning on their own — e.g. "the", "is", "a", "and", "of", "in".

Why Remove Stop Words?

• They occur in almost every document, so they add noise without adding discriminative signal for tasks like search, topic modeling, or classification.
• Removing them reduces vocabulary size and speeds up downstream processing (e.g. building a Bag-of-Words matrix in Unit 3).

Stop-word Removal with NLTK

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_words = set(stopwords.words('english'))
print(len(stop_words))     # 179 (approx, version-dependent)
print(list(stop_words)[:10])
# ['the', 'a', 'an', 'and', 'is', 'in', 'of', 'to', ...] (order varies)

text = "This is a simple example showing off stop word filtration."
tokens = word_tokenize(text)
filtered = [w for w in tokens if w.lower() not in stop_words]
print(filtered)
# ['simple', 'example', 'showing', 'stop', 'word', 'filtration', '.']

Stop-word Removal with spaCy

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a simple example showing off stop word filtration.")
filtered = [token.text for token in doc if not token.is_stop]
print(filtered)
# ['simple', 'example', 'showing', 'stop', 'word', 'filtration', '.']

Custom Stop-word Lists

Domain-specific stop words are often needed on top of (or instead of) the standard list.

custom_stopwords = set(stopwords.words('english'))
custom_stopwords.update(['also', 'said', 'would', 'could', 'us'])  # add domain-specific words
custom_stopwords.discard('not')  # KEEP "not" -- important for sentiment analysis!

text = "The product is not good, though it would still sell."
tokens = word_tokenize(text.lower())
filtered = [w for w in tokens if w not in custom_stopwords]
print(filtered)
# ['product', 'not', 'good', ',', 'still', 'sell', '.']

When NOT to Remove Stop Words

# Why blindly removing "not" breaks sentiment analysis:
text = "This movie is not good"
# After naive stop-word removal (if "not" is in the stop-word list):
# "movie good"  -- now reads as POSITIVE, the exact opposite of the original meaning!

The general rule: stop-word removal is a frequency-reduction technique for bag-of-words style tasks, not a universal preprocessing step — always evaluate its effect on your specific downstream task.