labs/Lab-4/Lab-4-Solution/02_Bag_of_Words_TFIDF_Solution.ipynb

Module 4 — Lab 2: Bag of Words and TF-IDF

Topics: CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage

Solution version: This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.

Learning objectives

After completing this lab, you will be able to:

• explain how Bag of Words maps documents to numeric vectors;
• inspect a document-term matrix and its vocabulary;
• fit TF-IDF on training text and transform unseen text;
• describe why text matrices are high-dimensional and sparse;
• compare unigram features with unigram-plus-bigram features.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.model_selection import train_test_split

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Make Bag of Words visible

Use three short messages so that every row, token, and count can be inspected directly.

In [ ]:

tiny_messages = [
    "free prize now",
    "free meeting today",
    "project meeting today",
]

count_vectorizer = CountVectorizer()
tiny_bow = count_vectorizer.fit_transform(tiny_messages)
tiny_bow_df = pd.DataFrame(
    tiny_bow.toarray(),
    columns=count_vectorizer.get_feature_names_out(),
    index=["message_1", "message_2", "message_3"],
)
tiny_bow_df

2. Prepare the UCI messages

Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]

conflicts = df.groupby("message")["label"].nunique()
conflict_texts = conflicts[conflicts.gt(1)].index
df = df[~df["message"].isin(conflict_texts)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)
print("Clean messages:", len(df))

3. Fit TF-IDF without leakage

fit_transform learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only transform.

In [ ]:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=RANDOM_STATE,
    stratify=y,
)

tfidf_vectorizer = TfidfVectorizer(lowercase=True)
X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)
X_test_tfidf = tfidf_vectorizer.transform(X_test)

density = X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1])
print("Training matrix shape:", X_train_tfidf.shape)
print("Test matrix shape:", X_test_tfidf.shape)
print("Training density:", f"{density:.4%}")

4. Inspect the learned feature space

A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.

In [ ]:

mean_tfidf = np.asarray(X_train_tfidf.mean(axis=0)).ravel()
top_terms = pd.Series(
    mean_tfidf,
    index=tfidf_vectorizer.get_feature_names_out(),
    name="mean_tfidf",
).sort_values(ascending=False).head(15)
top_terms

5. Add short phrases with bigrams

A unigram represents one token. A bigram can preserve short phrases such as call now or free entry, but it also increases the number of features.

In [ ]:

ngram_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1)
X_train_ngram = ngram_vectorizer.fit_transform(X_train)
ngram_features = ngram_vectorizer.get_feature_names_out()
example_bigrams = [term for term in ngram_features if " " in term][:10]

print("Unigram features:", X_train_tfidf.shape[1])
print("Unigram + bigram features:", X_train_ngram.shape[1])
print("Example bigrams:", example_bigrams)

Self-check

In [ ]:

assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))
assert tiny_bow_df.loc["message_1", "free"] == 1
assert len(X_train) + len(X_test) == len(df)
assert X_train_tfidf.shape[0] == len(X_train)
assert X_test_tfidf.shape[0] == len(X_test)
assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]
assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]
assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]
assert len(top_terms) == 15
print("✓ Lab 2 completed")

Suggested answers

1. Why is a text feature matrix usually sparse?
   Each message uses only a small fraction of the complete vocabulary, so most document-term cells are zero.
2. Why must the vectorizer be fitted only on training messages?
   Fitting on all messages reveals test-set vocabulary and frequency information during training, which is data leakage and can make evaluation optimistic.
3. What trade-off is introduced by adding bigrams?
   Bigrams preserve useful short phrases but expand the feature space, consume more memory, and can overfit when phrases are rare.