Levels of Language Processing
Human language can be analysed at several distinct levels, from the smallest meaningful unit up to full documents. An NLP system typically processes text through these levels in sequence.
| Level | Deals With | Example |
|---|---|---|
| Phonological | Sounds of speech (relevant for speech processing) | Pronunciation of "read" (present vs past) |
| Morphological | Structure of words — smallest meaning-bearing units | "unhappiness" = un + happy + ness |
| Lexical | Meaning of individual words (word-level) | "bank" has multiple senses |
| Syntactic | Grammatical structure of sentences | Subject-verb-object arrangement |
| Semantic | Meaning of sentences | "Colorless green ideas sleep furiously" is grammatical but meaningless |
| Discourse | Meaning across multiple sentences | Pronoun resolution across sentences |
| Pragmatic | Meaning in context of real-world use | Sarcasm, intent behind an utterance |
This lesson covers the first two: morphological and lexical analysis.
Morphological Analysis
Morphology studies the internal structure of words. A morpheme is the smallest unit of language that carries meaning — it cannot be broken down further without losing meaning.
| Term | Meaning | Example |
|---|---|---|
| Free morpheme | Can stand alone as a word | "happy", "run", "book" |
| Bound morpheme | Must attach to another morpheme | "un-", "-ness", "-ing", "-s" |
| Root/Stem | Core morpheme carrying the main meaning | "happy" in "unhappiness" |
| Prefix | Bound morpheme added before the root | "un-" in "unhappy" |
| Suffix | Bound morpheme added after the root | "-ness" in "happiness" |
word = "unhappiness"
# Morphological decomposition:
# un- (prefix, negation) + happy (root) + -ness (suffix, forms noun)
print("Root:", "happy")
print("Prefix:", "un-")
print("Suffix:", "-ness")Types of morphological processes:
- Inflection — modifies a word for grammar without changing its category: "walk" → "walked", "walks", "walking" (still a verb).
- Derivation — creates a new word, often changing category: "happy" (adjective) → "happiness" (noun).
- Compounding — combining two free morphemes: "note" + "book" = "notebook".
Lexical Analysis
Lexical analysis is the study of words and their meanings at the level of the lexicon (vocabulary). It involves:
- Identifying word tokens and their part of speech
- Mapping a word to its dictionary entry (lemma)
- Resolving word senses in context (e.g. "bat" — animal vs cricket bat)
import nltk
from nltk import pos_tag, word_tokenize
text = "The bank raised interest rates."
tokens = word_tokenize(text)
tagged = pos_tag(tokens)
print(tagged)
# [('The', 'DT'), ('bank', 'NN'), ('raised', 'VBD'),
# ('interest', 'NN'), ('rates', 'NNS'), ('.', '.')]Here, lexical analysis tells us "bank" is a noun (NN) — but it does not yet resolve which sense of "bank" is intended; that is the job of Word Sense Disambiguation, covered in Unit 2.