Introduction to Text Summarization

Text summarization automatically condenses a document into a shorter version that preserves its most important information. There are two fundamentally different approaches: extractive and abstractive.

Extractive vs Abstractive

Extractive Summarization — TF-IDF Sentence Scoring (Reusing Unit 3 Techniques)

The idea: score each sentence by summing the TF-IDF weight (recall the earlier lesson) of its words, then keep the top-scoring sentences.

from sklearn.feature_extraction.text import TfidfVectorizer
from nltk.tokenize import sent_tokenize
import numpy as np

document = """
Natural Language Processing enables computers to understand human language.
It combines linguistics, computer science, and artificial intelligence.
NLP powers applications like chatbots, translation, and search engines.
Preprocessing text is a critical first step in any NLP pipeline.
Many students find NLP challenging because of language ambiguity.
"""

sentences = sent_tokenize(document.strip())
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(sentences)

sentence_scores = np.array(tfidf_matrix.sum(axis=1)).flatten()
top_n = 2
top_indices = sentence_scores.argsort()[-top_n:][::-1]
top_indices.sort()   # keep original order

summary = " ".join([sentences[i] for i in top_indices])
print(summary)
# 'Natural Language Processing enables computers to understand human language.
#  NLP powers applications like chatbots, translation, and search engines.'

Extractive Summarization — TextRank Algorithm

TextRank is a graph-based ranking algorithm (inspired by Google's PageRank) applied to sentences: build a graph where sentences are nodes and edges are weighted by sentence similarity, then rank sentences by their graph centrality.

1. Split document into sentences
2. Build a similarity matrix between every pair of sentences
   (e.g. using cosine similarity of TF-IDF vectors)
3. Treat the similarity matrix as a graph
4. Run PageRank on this graph -> each sentence gets an importance score
5. Select the top-N highest-scoring sentences as the summary

import networkx as nx
from sklearn.metrics.pairwise import cosine_similarity

similarity_matrix = cosine_similarity(tfidf_matrix)
graph = nx.from_numpy_array(similarity_matrix)
scores = nx.pagerank(graph)

ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)
summary = " ".join([s for _, s in ranked_sentences[:2]])
print(summary)

Abstractive Summarization — Preview

Abstractive summarization requires a model that can generate novel text — this needs sequence-to-sequence architectures and, in modern systems, Transformer-based models (e.g. BART, T5, GPT-style LLMs) trained on large summarization datasets.

# Conceptual illustration (requires a pretrained transformer model, covered fully in Unit 4)
from transformers import pipeline

summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
result = summarizer(document, max_length=40, min_length=15, do_sample=False)
print(result[0]['summary_text'])
# "NLP enables computers to understand human language and powers
#  chatbots, translation, and search engines."

Evaluating Summaries — ROUGE Score

Summaries are commonly evaluated using ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which measures n-gram overlap between the generated summary and a human-written reference summary.

Extractive methods (TF-IDF scoring, TextRank) are fast, simple, and always grammatically correct — a solid default. Abstractive summarization, powered by the Transformer and LLM architectures covered in Unit 4, produces more natural, concise summaries but requires significantly more computational resources and training data.