labs/Lab-4/Lab-4-Practical/02_Bag_of_Words_TFIDF_Lab.ipynb

Module 4 — Lab 2: Bag of Words and TF-IDF

Topics: CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• explain how Bag of Words maps documents to numeric vectors;
• inspect a document-term matrix and its vocabulary;
• fit TF-IDF on training text and transform unseen text;
• describe why text matrices are high-dimensional and sparse;
• compare unigram features with unigram-plus-bigram features.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.model_selection import train_test_split

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Make Bag of Words visible

Use three short messages so that every row, token, and count can be inspected directly.

In [ ]:

tiny_messages = [
    "free prize now",
    "free meeting today",
    "project meeting today",
]

# TODO 1: fit a CountVectorizer on tiny_messages and transform the messages.
count_vectorizer = None
tiny_bow = None

# TODO 2: create a DataFrame whose columns are the learned feature names.
tiny_bow_df = None
tiny_bow_df

2. Prepare the UCI messages

Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]

conflicts = df.groupby("message")["label"].nunique()
conflict_texts = conflicts[conflicts.gt(1)].index
df = df[~df["message"].isin(conflict_texts)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)
print("Clean messages:", len(df))

3. Fit TF-IDF without leakage

fit_transform learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only transform.

In [ ]:

# TODO 3: split message and target into 80% training and 20% test data.
# Use random_state=42 and stratify=y.
X_train, X_test, y_train, y_test = None, None, None, None

# TODO 4: fit TfidfVectorizer on X_train only and transform X_test.
tfidf_vectorizer = None
X_train_tfidf = None
X_test_tfidf = None

print("Training matrix shape:", None)
print("Test matrix shape:", None)
print("Training density:", None)

4. Inspect the learned feature space

A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.

In [ ]:

# TODO 5: calculate each term's mean TF-IDF weight across training messages.
mean_tfidf = None

# TODO 6: build a descending Series and keep the 15 highest-weight terms.
top_terms = None
top_terms

5. Add short phrases with bigrams

A unigram represents one token. A bigram can preserve short phrases such as call now or free entry, but it also increases the number of features.

In [ ]:

# TODO 7: fit a second TF-IDF vectorizer with unigrams and bigrams.
ngram_vectorizer = None
X_train_ngram = None

print("Unigram features:", None)
print("Unigram + bigram features:", None)
print("Example bigrams:", None)

Self-check

In [ ]:

assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))
assert tiny_bow_df.loc["message_1", "free"] == 1
assert len(X_train) + len(X_test) == len(df)
assert X_train_tfidf.shape[0] == len(X_train)
assert X_test_tfidf.shape[0] == len(X_test)
assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]
assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]
assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]
assert len(top_terms) == 15
print("✓ Lab 2 completed")

Reflection questions

1. Why is a text feature matrix usually sparse?
2. Why must the vectorizer be fitted only on training messages?
3. What trade-off is introduced by adding bigrams?