labs/Lab-4/Lab-4-Practical/04_Linear_Model_Comparison_Lab.ipynb

Module 4 — Lab 4: Comparing Linear Text Classifiers

Topics: Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• compare several classifiers on one fixed train-test split;
• explain why linear decision boundaries work well with sparse TF-IDF features;
• choose a model using spam-sensitive metrics instead of accuracy alone;
• connect false positives and false negatives to operational consequences;
• explain why kernel SVM is not required for this classroom workflow.

In [1]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression, Perceptron
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Prepare one fixed experiment

Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.

In [2]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]
conflict_counts = df.groupby("message")["label"].nunique()
df = df[~df["message"].isin(conflict_counts[conflict_counts.gt(1)].index)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y
)

2. Define the candidate pipelines

All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.

In [3]:

# TODO 1: create four pipelines with identical TF-IDF settings.
# Models: MultinomialNB, Perceptron, LinearSVC, and LogisticRegression.
models = None

print("Models:", None)

Models: None

3. Train and evaluate fairly

Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses LinearSVC.

In [4]:

# TODO 2: fit every pipeline on the same training split.
# TODO 3: record accuracy, spam precision, spam recall, spam F1,
# false positives, and false negatives for each model.
fitted_models = None
predictions = None
comparison = None

comparison

4. Select and inspect the strongest model

For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.

In [5]:

# TODO 4: select the model with the highest spam F1.
best_model_name = None
best_pipeline = None
best_predictions = None

# TODO 5: create a DataFrame containing the best model's test errors.
best_errors = None

print("Best model by spam F1:", best_model_name)
print("Number of errors:", None)
best_errors.head(10)

Best model by spam F1: None
Number of errors: None

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 11
      9 print("Best model by spam F1:", best_model_name)
     10 print("Number of errors:", None)
---> 11 best_errors.head(10)

AttributeError: 'NoneType' object has no attribute 'head'

Interpretation checkpoint

A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.

Self-check

In [ ]:

assert set(models) == {"MultinomialNB", "Perceptron", "LinearSVC", "LogisticRegression"}
assert set(fitted_models) == set(models)
assert set(predictions) == set(models)
assert comparison.shape == (4, 6)
assert comparison.index.is_unique
assert comparison["spam_f1"].between(0, 1).all()
assert best_model_name in models
assert len(best_predictions) == len(y_test)
assert set(best_errors["error_type"].unique()).issubset({"false_positive", "false_negative"})
print("✓ Lab 4 completed")

Reflection questions

1. Why must every model use the same data split?
2. Why is a linear SVM appropriate for TF-IDF features?
3. Why might the highest-accuracy model still be unsuitable for spam filtering?