Regular Expressions for NLP
Regular expressions (regex) are patterns used to match, search, extract, and substitute text — an indispensable tool throughout the NLP preprocessing pipeline (custom tokenizers, cleaning, entity extraction).
Python's re Module — Core Functions
| Function | Purpose |
|---|---|
re.match() | Match only at the start of the string |
re.search() | Find the first match anywhere in the string |
re.findall() | Return all non-overlapping matches as a list |
re.sub() | Substitute matches with replacement text |
re.split() | Split a string wherever the pattern matches |
import re
text = "Contact us at support@sikshasarovar.com or admin@sikshasarovar.com"
# findall -- extract all email addresses
emails = re.findall(r'[\w.-]+@[\w.-]+\.\w+', text)
print(emails)
# ['support@sikshasarovar.com', 'admin@sikshasarovar.com']Common Regex Metacharacters
| Symbol | Meaning | Example | ||
|---|---|---|---|---|
. | Any character except newline | a.c matches "abc", "axc" | ||
\d | Any digit (0-9) | \d+ matches "2026" | ||
\w | Any word character (alphanumeric + _) | \w+ matches "hello_1" | ||
\s | Any whitespace | \s+ matches " " | ||
* | 0 or more of the preceding token | ab* matches "a", "ab", "abbb" | ||
+ | 1 or more | ab+ matches "ab", "abbb" (not "a") | ||
? | 0 or 1 (optional) | colou?r matches "color", "colour" | ||
^ | Start of string | ^Hi matches only if string starts with "Hi" | ||
$ | End of string | bye$ matches only if string ends with "bye" | ||
[] | Character class | [aeiou] matches any vowel | ||
{} | Exact repetition count | \d{10} matches exactly 10 digits | ||
| `\ | ` | Alternation (OR) | `cat\ | dog` matches "cat" or "dog" |
Practical NLP Regex Examples
import re
text = "Call me at 9876543210 or 011-23456789. Meeting on 05/08/2026."
# 1. Extract 10-digit phone numbers
phones = re.findall(r'\b\d{10}\b', text)
print(phones) # ['9876543210']
# 2. Extract dates in DD/MM/YYYY format
dates = re.findall(r'\d{2}/\d{2}/\d{4}', text)
print(dates) # ['05/08/2026']
# 3. Remove all digits
no_digits = re.sub(r'\d+', '', text)
print(no_digits)
# 4. Split text on multiple delimiters (comma, semicolon, period)
parts = re.split(r'[.,;]\s*', "NLP is fun, powerful; and useful.")
print(parts) # ['NLP is fun', 'powerful', 'and useful', '']Building a Custom Tokenizer with Regex
import re
def custom_tokenize(text):
# keep words, and standalone punctuation, drop everything else
return re.findall(r"[A-Za-z]+(?:'[A-Za-z]+)?|[.,!?;]", text)
print(custom_tokenize("Isn't NLP amazing? Yes, it is!"))
# ["Isn't", 'NLP', 'amazing', '?', 'Yes', ',', 'it', 'is', '!']Cleaning Social Media Text with Regex
import re
tweet = "OMG this movie is amazing!!! 😍 #MustWatch @filmcritic https://example.com/review"
no_urls = re.sub(r'http\S+', '', tweet)
no_mentions = re.sub(r'@\w+', '', no_urls)
no_hashtag_symbol = re.sub(r'#(\w+)', r'\1', no_mentions) # keep the word, drop '#'
cleaned = re.sub(r'\s+', ' ', no_hashtag_symbol).strip()
print(cleaned)
# OMG this movie is amazing!!! 😍 MustWatchRegular expressions give you fine-grained, rule-based control that complements the statistical/ML-based methods covered in later units — many real-world NLP pipelines still rely on regex for cleaning and rule-based extraction.