Feature Extraction — Introduction, Need & Importance

Machine learning algorithms (Naïve Bayes, SVM, logistic regression, neural networks) cannot operate on raw text directly — they require numeric input. Feature extraction is the process of converting cleaned, preprocessed text (Unit 1) into numeric vectors that capture meaningful properties of the text.

Why Feature Extraction Is Needed

1. ML models are mathematical functions — they operate on numbers (vectors, matrices), not strings.
2. Text has variable length — "good" and "this movie was absolutely fantastic" have different lengths, but a model needs fixed-size input.
3. Words need a numeric representation of meaning/importance — some words matter more than others for a given task.

The General Feature Extraction Pipeline

"The movie was great" -> tokenize -> ["movie", "great"] (after stop-word removal)
                       -> vectorize -> [0, 1, 0, 1, 0, 0, ...]   (fixed-length numeric vector)

Categories of Text Features

Importance of Good Feature Extraction

• Directly determines model performance — even the best classifier cannot learn from poorly chosen features.
• Controls dimensionality — raw vocabulary can be 100,000+ words; feature extraction techniques manage this (e.g. TF-IDF weighting, dimensionality reduction).
• Encodes domain knowledge — e.g. keeping "not" as a feature (Unit 1) is crucial for sentiment tasks.
• Impacts computational cost — sparse, high-dimensional features (BoW) vs dense, compact ones (embeddings) trade off differently in memory and speed.

A Minimal End-to-End Example

from sklearn.feature_extraction.text import CountVectorizer

documents = [
    "The movie was great",
    "The movie was terrible",
    "Great acting and great story"
]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)

print(vectorizer.get_feature_names_out())
# ['acting' 'and' 'great' 'movie' 'story' 'terrible' 'the' 'was']
print(X.toarray())
# [[0 0 1 1 0 0 1 1]
#  [0 0 0 1 0 1 1 1]
#  [1 1 2 0 1 0 0 0]]

Each document is now a fixed-length numeric vector — ready to be fed into any classical ML algorithm. This lesson sets up the "why"; the next two lessons cover the two most important classical feature extraction techniques in depth: Bag of Words and TF-IDF.