labs/Lab-4/Lab-4-Practical/06_Capstone_SMS_Spam_Detector_Lab.ipynb

Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector

Topics: complete workflow, Logistic Regression, actions, error analysis, reporting

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Mission

Build one complete, reproducible, and leakage-safe spam-detection workflow:

1. load and audit the UCI SMS Spam Collection;
2. create one stratified holdout test set;
3. train a word-level TF-IDF and Logistic Regression pipeline;
4. evaluate spam-sensitive metrics and operational errors;
5. inspect influential terms and test custom messages;
6. export evidence and an analyst-facing summary.

The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.

In [ ]:

from pathlib import Path
import textwrap
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    ConfusionMatrixDisplay,
    accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path("data/SMSSpamCollection")
OUTPUT_DIR = Path("output")
RANDOM_STATE = 42
DECISION_THRESHOLD = 0.50

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

1. Load, audit, and split the data

Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.

In [ ]:

# TODO 1: load the UCI file, normalize the two columns, remove invalid
# rows and conflicts, remove exact duplicates, and create target.
raw_df = None
clean_df = None

# TODO 2: create X and y, then make a stratified 80/20 split.
X = None
y = None
X_train, X_test, y_train, y_test = None, None, None, None

print("Raw rows:", None)
print("Clean rows:", None)
print("Training / test:", None)

2. Build and train the final pipeline

The pipeline learns its vocabulary and IDF weights from the training messages only.

In [ ]:

# TODO 3: create a word unigram+bigram TF-IDF and Logistic Regression pipeline.
final_pipeline = None

# TODO 4: fit on training data and calculate test probabilities and predictions.
test_probabilities = None
test_predictions = None

print("Predictions created:", None)

3. Evaluate the held-out test set

Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.

In [ ]:

# TODO 5: calculate the confusion matrix and capstone metrics.
cm = None
tn, fp, fn, tp = None, None, None, None
metrics = None

print(pd.Series(metrics).round(4))
print(None)

4. Create an analyst error table

Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.

In [ ]:

# TODO 6: create prediction_table with message, actual_label,
# predicted_label, spam_probability, action, and error_type.
prediction_table = None

# TODO 7: keep only false positives and false negatives in error_table.
error_table = None

print("Errors by type:")
print(None)
error_table.head(10)

5. Inspect influential words and phrases

Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.

In [ ]:

# TODO 8: extract the 20 strongest spam terms and 20 strongest ham terms.
top_terms = None
top_terms.head(10)

6. Challenge the detector with new messages

The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as w1nner or FR33, which exposes a robustness limitation and motivates an optional character n-gram experiment.

In [ ]:

custom_messages = pd.Series([
    "Can we move the project meeting to 3 pm?",
    "Congratulations! Claim your free cash prize now",
    "URGENT: verify your account at the link below",
    "You are a w1nner, reply FR33 to collect your reward",
    "Please call me when you arrive at the station",
], name="message")

# TODO 9: score the custom messages and assign the three policy actions.
custom_probabilities = None
custom_results = None
custom_results

7. Export evidence and write the analyst summary

In [ ]:

# TODO 10: save metrics, predictions, top terms, and a confusion-matrix chart.
# TODO 11: write an analyst summary containing the dataset size, spam rate,
# threshold, four primary metrics, FP/FN counts, and one limitation.
summary = None

print(summary)

Self-check

In [ ]:

expected_files = {
    "capstone_metrics.csv",
    "capstone_predictions.csv",
    "capstone_top_terms.csv",
    "capstone_confusion_matrix.png",
    "capstone_summary.txt",
}

assert set(clean_df["target"].unique()) == {0, 1}
assert not clean_df.duplicated(subset=["label", "message"]).any()
assert abs(y_train.mean() - y_test.mean()) < 0.02
assert list(final_pipeline.named_steps) == ["tfidf", "model"]
assert len(test_predictions) == len(y_test)
assert cm.shape == (2, 2)
assert (tn + fp + fn + tp) == len(y_test)
assert len(prediction_table) == len(y_test)
assert set(prediction_table["action"].unique()).issubset({"allow", "analyst_review", "quarantine"})
assert len(top_terms) == 40
assert len(custom_results) == 5
assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})
assert "Limitation" in summary
print("✓ Lab 6 completed")

Reflection questions

1. What do the false positives and false negatives mean to a security team?
2. Why should the custom obfuscated message be included in the capstone?
3. What evidence is still missing before this classroom model could support real email security decisions?

In [ ]:

In [ ]: