labs/Lab-4/Lab-4-Solution/06_Capstone_SMS_Spam_Detector_Solution.ipynb

Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector

Topics: complete workflow, Logistic Regression, actions, error analysis, reporting

Solution version: This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.

Mission

Build one complete, reproducible, and leakage-safe spam-detection workflow:

1. load and audit the UCI SMS Spam Collection;
2. create one stratified holdout test set;
3. train a word-level TF-IDF and Logistic Regression pipeline;
4. evaluate spam-sensitive metrics and operational errors;
5. inspect influential terms and test custom messages;
6. export evidence and an analyst-facing summary.

The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.

In [ ]:

from pathlib import Path
import textwrap
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    ConfusionMatrixDisplay,
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path("data/SMSSpamCollection")
OUTPUT_DIR = Path("output")
RANDOM_STATE = 42
DECISION_THRESHOLD = 0.50

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

1. Load, audit, and split the data

Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.

In [ ]:

raw_df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
clean_df = raw_df.copy()
clean_df["label"] = clean_df["label"].astype("string").str.strip().str.lower()
clean_df["message"] = clean_df["message"].astype("string").str.strip()
clean_df = clean_df.dropna(subset=["label", "message"])
clean_df = clean_df[
    clean_df["label"].isin(["ham", "spam"]) & clean_df["message"].ne("")
]
conflict_counts = clean_df.groupby("message")["label"].nunique()
conflict_texts = conflict_counts[conflict_counts.gt(1)].index
clean_df = clean_df[~clean_df["message"].isin(conflict_texts)]
clean_df = clean_df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)
clean_df["target"] = clean_df["label"].map({"ham": 0, "spam": 1}).astype(int)

X = clean_df["message"]
y = clean_df["target"]
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=RANDOM_STATE,
    stratify=y,
)

print("Raw rows:", len(raw_df))
print("Clean rows:", len(clean_df))
print("Training / test:", len(X_train), len(X_test))

2. Build and train the final pipeline

The pipeline learns its vocabulary and IDF weights from the training messages only.

In [ ]:

final_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(
        lowercase=True,
        ngram_range=(1, 2),
        min_df=1,
        sublinear_tf=True,
    )),
    ("model", LogisticRegression(
        max_iter=2000,
        random_state=RANDOM_STATE,
    )),
])
final_pipeline.fit(X_train, y_train)
test_probabilities = final_pipeline.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= DECISION_THRESHOLD).astype(int)

print("Predictions created:", len(test_predictions))

3. Evaluate the held-out test set

Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.

In [ ]:

cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])
tn, fp, fn, tp = cm.ravel()
metrics = {
    "decision_threshold": DECISION_THRESHOLD,
    "accuracy": accuracy_score(y_test, test_predictions),
    "spam_precision": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),
    "spam_recall": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),
    "spam_f1": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),
    "true_negatives": int(tn),
    "false_positives": int(fp),
    "false_negatives": int(fn),
    "true_positives": int(tp),
}

print(pd.Series(metrics).round(4))
print(classification_report(
    y_test,
    test_predictions,
    target_names=["ham", "spam"],
    zero_division=0,
))

4. Create an analyst error table

Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.

In [ ]:

actions = pd.cut(
    test_probabilities,
    bins=[-np.inf, 0.30, 0.70, np.inf],
    labels=["allow", "analyst_review", "quarantine"],
    right=False,
).astype("string")

prediction_table = pd.DataFrame({
    "message": X_test.to_numpy(),
    "actual_label": np.where(y_test.to_numpy() == 1, "spam", "ham"),
    "predicted_label": np.where(test_predictions == 1, "spam", "ham"),
    "spam_probability": test_probabilities,
    "action": actions,
})
prediction_table["error_type"] = np.select(
    [
        (prediction_table["actual_label"] == "ham") & (prediction_table["predicted_label"] == "spam"),
        (prediction_table["actual_label"] == "spam") & (prediction_table["predicted_label"] == "ham"),
    ],
    ["false_positive", "false_negative"],
    default="correct",
)
error_table = prediction_table[prediction_table["error_type"].ne("correct")].copy()

print("Errors by type:")
print(error_table["error_type"].value_counts())
error_table.head(10)

5. Inspect influential words and phrases

Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.

In [ ]:

feature_names = final_pipeline.named_steps["tfidf"].get_feature_names_out()
coefficients = pd.Series(
    final_pipeline.named_steps["model"].coef_[0],
    index=feature_names,
    name="coefficient",
)

top_spam = coefficients.nlargest(20).reset_index()
top_spam.columns = ["term", "coefficient"]
top_spam["direction"] = "spam"
top_ham = coefficients.nsmallest(20).reset_index()
top_ham.columns = ["term", "coefficient"]
top_ham["direction"] = "ham"
top_terms = pd.concat([top_spam, top_ham], ignore_index=True)
top_terms.head(10)

6. Challenge the detector with new messages

The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as w1nner or FR33, which exposes a robustness limitation and motivates an optional character n-gram experiment.

In [ ]:

custom_messages = pd.Series([
    "Can we move the project meeting to 3 pm?",
    "Congratulations! Claim your free cash prize now",
    "URGENT: verify your account at the link below",
    "You are a w1nner, reply FR33 to collect your reward",
    "Please call me when you arrive at the station",
], name="message")

custom_probabilities = final_pipeline.predict_proba(custom_messages)[:, 1]
custom_results = pd.DataFrame({
    "message": custom_messages,
    "spam_probability": custom_probabilities,
    "predicted_label": np.where(custom_probabilities >= DECISION_THRESHOLD, "spam", "ham"),
    "action": pd.cut(
        custom_probabilities,
        bins=[-np.inf, 0.30, 0.70, np.inf],
        labels=["allow", "analyst_review", "quarantine"],
        right=False,
    ).astype("string"),
})
custom_results

7. Export evidence and write the analyst summary

In [ ]:

pd.Series(metrics, name="value").to_csv(OUTPUT_DIR / "capstone_metrics.csv", header=True)
prediction_table.to_csv(OUTPUT_DIR / "capstone_predictions.csv", index=False)
top_terms.to_csv(OUTPUT_DIR / "capstone_top_terms.csv", index=False)

fig, ax = plt.subplots(figsize=(5, 4))
ConfusionMatrixDisplay(
    confusion_matrix=cm,
    display_labels=["ham", "spam"],
).plot(ax=ax, cmap="Blues", colorbar=False)
ax.set_title("SMS Spam Detector — Test Set")
fig.tight_layout()
fig.savefig(OUTPUT_DIR / "capstone_confusion_matrix.png", dpi=160, bbox_inches="tight")
plt.close(fig)

summary = f'''SMS Spam Detector — Analyst Summary
===================================
Source dataset: UCI SMS Spam Collection
Clean rows: {len(clean_df)}
Training rows: {len(X_train)}
Test rows: {len(X_test)}
Overall spam rate: {y.mean():.2%}
Decision threshold: {DECISION_THRESHOLD:.2f}

Test performance
----------------
Accuracy: {metrics['accuracy']:.3f}
Spam precision: {metrics['spam_precision']:.3f}
Spam recall: {metrics['spam_recall']:.3f}
Spam F1: {metrics['spam_f1']:.3f}
False positives: {fp}
False negatives: {fn}

Operational interpretation
--------------------------
Messages below 0.30 are allowed, scores from 0.30 to below 0.70 are sent
for analyst review, and scores of 0.70 or more are quarantined.

Limitation
----------
This classroom model uses SMS body text only. It does not inspect email headers,
sender domains, attachments, destination URLs, or changing attacker behavior.
'''
summary = textwrap.dedent(summary).strip() + "\n"
(OUTPUT_DIR / "capstone_summary.txt").write_text(summary, encoding="utf-8")

print(summary)

Self-check

In [ ]:

expected_files = {
    "capstone_metrics.csv",
    "capstone_predictions.csv",
    "capstone_top_terms.csv",
    "capstone_confusion_matrix.png",
    "capstone_summary.txt",
}

assert set(clean_df["target"].unique()) == {0, 1}
assert not clean_df.duplicated(subset=["label", "message"]).any()
assert abs(y_train.mean() - y_test.mean()) < 0.02
assert list(final_pipeline.named_steps) == ["tfidf", "model"]
assert len(test_predictions) == len(y_test)
assert cm.shape == (2, 2)
assert (tn + fp + fn + tp) == len(y_test)
assert len(prediction_table) == len(y_test)
assert set(prediction_table["action"].unique()).issubset({"allow", "analyst_review", "quarantine"})
assert len(top_terms) == 40
assert len(custom_results) == 5
assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})
assert "Limitation" in summary
print("✓ Lab 6 completed")

Suggested answers

1. What do the false positives and false negatives mean to a security team?
   False positives consume review effort or block legitimate communication. False negatives allow spam through and may expose users to fraud or malicious links. The acceptable balance depends on organizational cost and risk.
2. Why should the custom obfuscated message be included in the capstone?
   It checks whether the learned word vocabulary is brittle when attackers alter spelling. A failure is useful evidence for considering character n-grams or additional signals later.
3. What evidence is still missing before this classroom model could support real email security decisions?
   The model needs representative current email data, sender and header features, URL and attachment analysis, temporal validation, calibrated policy costs, adversarial testing, drift monitoring, privacy review, and human oversight.