labs/Lab-4/Lab-4-Solution/04_Linear_Model_Comparison_Solution.ipynb

Module 4 — Lab 4: Comparing Linear Text Classifiers

Topics: Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison

Solution version: This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.

Learning objectives

After completing this lab, you will be able to:

• compare several classifiers on one fixed train-test split;
• explain why linear decision boundaries work well with sparse TF-IDF features;
• choose a model using spam-sensitive metrics instead of accuracy alone;
• connect false positives and false negatives to operational consequences;
• explain why kernel SVM is not required for this classroom workflow.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression, Perceptron
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Prepare one fixed experiment

Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]
conflict_counts = df.groupby("message")["label"].nunique()
df = df[~df["message"].isin(conflict_counts[conflict_counts.gt(1)].index)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y
)

2. Define the candidate pipelines

All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.

In [ ]:

def make_pipeline(model):
    return Pipeline([
        ("tfidf", TfidfVectorizer(
            lowercase=True,
            ngram_range=(1, 2),
            min_df=1,
            sublinear_tf=True,
        )),
        ("model", model),
    ])


models = {
    "MultinomialNB": make_pipeline(MultinomialNB(alpha=1.0)),
    "Perceptron": make_pipeline(Perceptron(
        max_iter=2000,
        tol=1e-3,
        random_state=RANDOM_STATE,
    )),
    "LinearSVC": make_pipeline(LinearSVC(random_state=RANDOM_STATE)),
    "LogisticRegression": make_pipeline(LogisticRegression(
        max_iter=2000,
        random_state=RANDOM_STATE,
    )),
}

print("Models:", list(models))

3. Train and evaluate fairly

Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses LinearSVC.

In [ ]:

fitted_models = {}
predictions = {}
rows = []

for model_name, pipeline in models.items():
    pipeline.fit(X_train, y_train)
    y_pred = pipeline.predict(X_test)
    cm = confusion_matrix(y_test, y_pred, labels=[0, 1])
    tn, fp, fn, tp = cm.ravel()

    fitted_models[model_name] = pipeline
    predictions[model_name] = y_pred
    rows.append({
        "model": model_name,
        "accuracy": accuracy_score(y_test, y_pred),
        "spam_precision": precision_score(y_test, y_pred, pos_label=1, zero_division=0),
        "spam_recall": recall_score(y_test, y_pred, pos_label=1, zero_division=0),
        "spam_f1": f1_score(y_test, y_pred, pos_label=1, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
    })

comparison = (
    pd.DataFrame(rows)
    .set_index("model")
    .sort_values(["spam_f1", "spam_recall"], ascending=False)
)
comparison.round(4)

4. Select and inspect the strongest model

For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.

In [ ]:

best_model_name = comparison["spam_f1"].idxmax()
best_pipeline = fitted_models[best_model_name]
best_predictions = predictions[best_model_name]

best_errors = pd.DataFrame({
    "message": X_test.to_numpy(),
    "actual": y_test.to_numpy(),
    "predicted": best_predictions,
})
best_errors = best_errors[best_errors["actual"].ne(best_errors["predicted"])].copy()
best_errors["error_type"] = np.where(
    (best_errors["actual"] == 0) & (best_errors["predicted"] == 1),
    "false_positive",
    "false_negative",
)

print("Best model by spam F1:", best_model_name)
print("Number of errors:", len(best_errors))
best_errors.head(10)

Interpretation checkpoint

A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.

Self-check

In [ ]:

assert set(models) == {"MultinomialNB", "Perceptron", "LinearSVC", "LogisticRegression"}
assert set(fitted_models) == set(models)
assert set(predictions) == set(models)
assert comparison.shape == (4, 6)
assert comparison.index.is_unique
assert comparison["spam_f1"].between(0, 1).all()
assert best_model_name in models
assert len(best_predictions) == len(y_test)
assert set(best_errors["error_type"].unique()).issubset({"false_positive", "false_negative"})
print("✓ Lab 4 completed")

Suggested answers

1. Why must every model use the same data split?
   Changing both the split and model would confound the comparison. A fixed split isolates the effect of the classifier and makes the scores directly comparable.
2. Why is a linear SVM appropriate for TF-IDF features?
   TF-IDF creates many sparse features. Linear SVM can learn a separating hyperplane efficiently in this high-dimensional space without calculating an expensive nonlinear kernel matrix.
3. Why might the highest-accuracy model still be unsuitable for spam filtering?
   Accuracy can be dominated by the majority ham class. A model may still miss too much spam or block too many legitimate messages, which is visible in spam recall, precision, and error counts.