labs/Lab-4/Lab-4-Practical/03_Naive_Bayes_Baseline_Lab.ipynb

Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline

Topics: Pipeline, MultinomialNB, confusion matrix, baseline, error analysis

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• place TF-IDF and Multinomial Naive Bayes in one pipeline;
• explain how the pipeline prevents preprocessing leakage;
• calculate spam precision, recall, and F1;
• compare a trained classifier with a naive baseline;
• inspect false positives and false negatives.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Load, audit, and split the messages

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]
conflict_counts = df.groupby("message")["label"].nunique()
df = df[~df["message"].isin(conflict_counts[conflict_counts.gt(1)].index)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y
)
print("Training messages:", len(X_train))
print("Test messages:", len(X_test))
print("Training spam rate:", f"{y_train.mean():.2%}")
print("Test spam rate:", f"{y_test.mean():.2%}")

2. Build and train one pipeline

The pipeline fits TF-IDF only on X_train. During prediction, it applies the already-fitted transformation to X_test.

In [ ]:

# TODO 1: create a Pipeline named nb_pipeline with a TF-IDF vectorizer
# followed by MultinomialNB(alpha=1.0).
nb_pipeline = None

# TODO 2: fit the pipeline on X_train and y_train, then predict X_test.
y_pred = None

print(nb_pipeline)

3. Evaluate the spam class

• Spam precision: when the model blocks a message, how often is it truly spam?
• Spam recall: of all spam messages, how many did the model catch?
• Spam F1: harmonic mean of spam precision and recall.

In [ ]:

# TODO 3: calculate the confusion matrix and unpack tn, fp, fn, tp.
cm = None
tn, fp, fn, tp = None, None, None, None

# TODO 4: calculate accuracy and spam precision, recall, and F1.
nb_metrics = None

print(pd.Series(nb_metrics).round(4))
print("\nClassification report:")
print(None)

4. Compare with a naive baseline

Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.

In [ ]:

# TODO 5: train a most_frequent DummyClassifier and predict X_test.
dummy = None
dummy_pred = None

# TODO 6: compare the two models using spam F1 and accuracy.
comparison = None
comparison

5. Inspect operational errors

A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.

In [ ]:

# TODO 7: create a DataFrame with message, actual, and predicted columns.
error_analysis = None

# TODO 8: keep only incorrect predictions and add an error_type column
# containing false_positive or false_negative.
errors = None

print(errors["error_type"].value_counts())
errors.head(10)

Self-check

In [ ]:

assert list(nb_pipeline.named_steps) == ["tfidf", "model"]
assert isinstance(nb_pipeline.named_steps["model"], MultinomialNB)
assert len(y_pred) == len(y_test)
assert cm.shape == (2, 2)
assert (tn + fp + fn + tp) == len(y_test)
assert set(nb_metrics) == {"accuracy", "spam_precision", "spam_recall", "spam_f1"}
assert comparison.loc["Most-frequent baseline", "spam_f1"] == 0
assert set(errors["error_type"].unique()).issubset({"false_positive", "false_negative"})
print("✓ Lab 3 completed")

Reflection questions

1. Why is the most-frequent classifier an important baseline?
2. Which operational harm is represented by a false positive?
3. Why should TF-IDF and the model be placed in one pipeline?