labs/Lab-4/Lab-4-Solution/03_Naive_Bayes_Baseline_Solution.ipynb

Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline

Topics: Pipeline, MultinomialNB, confusion matrix, baseline, error analysis

Solution version: This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.

Learning objectives

After completing this lab, you will be able to:

• place TF-IDF and Multinomial Naive Bayes in one pipeline;
• explain how the pipeline prevents preprocessing leakage;
• calculate spam precision, recall, and F1;
• compare a trained classifier with a naive baseline;
• inspect false positives and false negatives.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Load, audit, and split the messages

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]
conflict_counts = df.groupby("message")["label"].nunique()
df = df[~df["message"].isin(conflict_counts[conflict_counts.gt(1)].index)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y
)
print("Training messages:", len(X_train))
print("Test messages:", len(X_test))
print("Training spam rate:", f"{y_train.mean():.2%}")
print("Test spam rate:", f"{y_test.mean():.2%}")

2. Build and train one pipeline

The pipeline fits TF-IDF only on X_train. During prediction, it applies the already-fitted transformation to X_test.

In [ ]:

nb_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(lowercase=True, ngram_range=(1, 1))),
    ("model", MultinomialNB(alpha=1.0)),
])
nb_pipeline.fit(X_train, y_train)
y_pred = nb_pipeline.predict(X_test)

print(nb_pipeline)

3. Evaluate the spam class

• Spam precision: when the model blocks a message, how often is it truly spam?
• Spam recall: of all spam messages, how many did the model catch?
• Spam F1: harmonic mean of spam precision and recall.

In [ ]:

cm = confusion_matrix(y_test, y_pred, labels=[0, 1])
tn, fp, fn, tp = cm.ravel()

nb_metrics = {
    "accuracy": accuracy_score(y_test, y_pred),
    "spam_precision": precision_score(y_test, y_pred, pos_label=1, zero_division=0),
    "spam_recall": recall_score(y_test, y_pred, pos_label=1, zero_division=0),
    "spam_f1": f1_score(y_test, y_pred, pos_label=1, zero_division=0),
}

print(pd.Series(nb_metrics).round(4))
print("\nClassification report:")
print(classification_report(y_test, y_pred, target_names=["ham", "spam"], zero_division=0))

4. Compare with a naive baseline

Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.

In [ ]:

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train.to_frame(), y_train)
dummy_pred = dummy.predict(X_test.to_frame())

comparison = pd.DataFrame([
    {
        "model": "Most-frequent baseline",
        "accuracy": accuracy_score(y_test, dummy_pred),
        "spam_f1": f1_score(y_test, dummy_pred, pos_label=1, zero_division=0),
    },
    {
        "model": "TF-IDF + MultinomialNB",
        "accuracy": nb_metrics["accuracy"],
        "spam_f1": nb_metrics["spam_f1"],
    },
]).set_index("model")
comparison

5. Inspect operational errors

A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.

In [ ]:

error_analysis = pd.DataFrame({
    "message": X_test.to_numpy(),
    "actual": y_test.to_numpy(),
    "predicted": y_pred,
})
errors = error_analysis[error_analysis["actual"].ne(error_analysis["predicted"])].copy()
errors["error_type"] = np.where(
    (errors["actual"] == 0) & (errors["predicted"] == 1),
    "false_positive",
    "false_negative",
)

print(errors["error_type"].value_counts())
errors.head(10)

Self-check

In [ ]:

assert list(nb_pipeline.named_steps) == ["tfidf", "model"]
assert isinstance(nb_pipeline.named_steps["model"], MultinomialNB)
assert len(y_pred) == len(y_test)
assert cm.shape == (2, 2)
assert (tn + fp + fn + tp) == len(y_test)
assert set(nb_metrics) == {"accuracy", "spam_precision", "spam_recall", "spam_f1"}
assert comparison.loc["Most-frequent baseline", "spam_f1"] == 0
assert set(errors["error_type"].unique()).issubset({"false_positive", "false_negative"})
print("✓ Lab 3 completed")

Suggested answers

1. Why is the most-frequent classifier an important baseline?
   It reveals how much accuracy comes from class imbalance alone. A useful spam detector must outperform the baseline on spam-sensitive metrics.
2. Which operational harm is represented by a false positive?
   A legitimate message is incorrectly blocked or quarantined, which can interrupt communication and reduce user trust.
3. Why should TF-IDF and the model be placed in one pipeline?
   The pipeline guarantees that vocabulary and IDF statistics are learned only during fitting and then reused consistently, reducing leakage and preprocessing mistakes.