{ "cells": [ { "cell_type": "markdown", "id": "4116df5d", "metadata": {}, "source": [ "# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n", "\n", " \n", "**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n", "\n", "> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n" ] }, { "cell_type": "markdown", "id": "81a0abfa", "metadata": {}, "source": [ "## Mission\n", "\n", "Build one complete, reproducible, and leakage-safe spam-detection workflow:\n", "\n", "1. load and audit the UCI SMS Spam Collection;\n", "2. create one stratified holdout test set;\n", "3. train a word-level TF-IDF and Logistic Regression pipeline;\n", "4. evaluate spam-sensitive metrics and operational errors;\n", "5. inspect influential terms and test custom messages;\n", "6. export evidence and an analyst-facing summary.\n", "\n", "The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b945ce17", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import textwrap\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import (\n", " ConfusionMatrixDisplay,\n", " accuracy_score,\n", " classification_report,\n", " confusion_matrix,\n", " f1_score,\n", " precision_score,\n", " recall_score,\n", ")\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import Pipeline\n", "\n", "DATA_PATH = Path(\"data/SMSSpamCollection\")\n", "OUTPUT_DIR = Path(\"output\")\n", "RANDOM_STATE = 42\n", "DECISION_THRESHOLD = 0.50\n", "\n", "if not DATA_PATH.exists():\n", " raise FileNotFoundError(\n", " \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n", " )\n", "\n", "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n" ] }, { "cell_type": "markdown", "id": "8ea5e886", "metadata": {}, "source": [ "## 1. Load, audit, and split the data\n", "\n", "Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b604f8da", "metadata": {}, "outputs": [], "source": [ "raw_df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n", "clean_df = raw_df.copy()\n", "clean_df[\"label\"] = clean_df[\"label\"].astype(\"string\").str.strip().str.lower()\n", "clean_df[\"message\"] = clean_df[\"message\"].astype(\"string\").str.strip()\n", "clean_df = clean_df.dropna(subset=[\"label\", \"message\"])\n", "clean_df = clean_df[\n", " clean_df[\"label\"].isin([\"ham\", \"spam\"]) & clean_df[\"message\"].ne(\"\")\n", "]\n", "conflict_counts = clean_df.groupby(\"message\")[\"label\"].nunique()\n", "conflict_texts = conflict_counts[conflict_counts.gt(1)].index\n", "clean_df = clean_df[~clean_df[\"message\"].isin(conflict_texts)]\n", "clean_df = clean_df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n", "clean_df[\"target\"] = clean_df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n", "\n", "X = clean_df[\"message\"]\n", "y = clean_df[\"target\"]\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X,\n", " y,\n", " test_size=0.20,\n", " random_state=RANDOM_STATE,\n", " stratify=y,\n", ")\n", "\n", "print(\"Raw rows:\", len(raw_df))\n", "print(\"Clean rows:\", len(clean_df))\n", "print(\"Training / test:\", len(X_train), len(X_test))\n" ] }, { "cell_type": "markdown", "id": "578afb0e", "metadata": {}, "source": [ "## 2. Build and train the final pipeline\n", "\n", "The pipeline learns its vocabulary and IDF weights from the training messages only.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b9eafd46", "metadata": {}, "outputs": [], "source": [ "final_pipeline = Pipeline([\n", " (\"tfidf\", TfidfVectorizer(\n", " lowercase=True,\n", " ngram_range=(1, 2),\n", " min_df=1,\n", " sublinear_tf=True,\n", " )),\n", " (\"model\", LogisticRegression(\n", " max_iter=2000,\n", " random_state=RANDOM_STATE,\n", " )),\n", "])\n", "final_pipeline.fit(X_train, y_train)\n", "test_probabilities = final_pipeline.predict_proba(X_test)[:, 1]\n", "test_predictions = (test_probabilities >= DECISION_THRESHOLD).astype(int)\n", "\n", "print(\"Predictions created:\", len(test_predictions))\n" ] }, { "cell_type": "markdown", "id": "7b6ed943", "metadata": {}, "source": [ "## 3. Evaluate the held-out test set\n", "\n", "Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7faddf17", "metadata": {}, "outputs": [], "source": [ "cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n", "tn, fp, fn, tp = cm.ravel()\n", "metrics = {\n", " \"decision_threshold\": DECISION_THRESHOLD,\n", " \"accuracy\": accuracy_score(y_test, test_predictions),\n", " \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n", " \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n", " \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n", " \"true_negatives\": int(tn),\n", " \"false_positives\": int(fp),\n", " \"false_negatives\": int(fn),\n", " \"true_positives\": int(tp),\n", "}\n", "\n", "print(pd.Series(metrics).round(4))\n", "print(classification_report(\n", " y_test,\n", " test_predictions,\n", " target_names=[\"ham\", \"spam\"],\n", " zero_division=0,\n", "))\n" ] }, { "cell_type": "markdown", "id": "57c486cb", "metadata": {}, "source": [ "## 4. Create an analyst error table\n", "\n", "Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d612fce7", "metadata": {}, "outputs": [], "source": [ "actions = pd.cut(\n", " test_probabilities,\n", " bins=[-np.inf, 0.30, 0.70, np.inf],\n", " labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n", " right=False,\n", ").astype(\"string\")\n", "\n", "prediction_table = pd.DataFrame({\n", " \"message\": X_test.to_numpy(),\n", " \"actual_label\": np.where(y_test.to_numpy() == 1, \"spam\", \"ham\"),\n", " \"predicted_label\": np.where(test_predictions == 1, \"spam\", \"ham\"),\n", " \"spam_probability\": test_probabilities,\n", " \"action\": actions,\n", "})\n", "prediction_table[\"error_type\"] = np.select(\n", " [\n", " (prediction_table[\"actual_label\"] == \"ham\") & (prediction_table[\"predicted_label\"] == \"spam\"),\n", " (prediction_table[\"actual_label\"] == \"spam\") & (prediction_table[\"predicted_label\"] == \"ham\"),\n", " ],\n", " [\"false_positive\", \"false_negative\"],\n", " default=\"correct\",\n", ")\n", "error_table = prediction_table[prediction_table[\"error_type\"].ne(\"correct\")].copy()\n", "\n", "print(\"Errors by type:\")\n", "print(error_table[\"error_type\"].value_counts())\n", "error_table.head(10)\n" ] }, { "cell_type": "markdown", "id": "99d1bba1", "metadata": {}, "source": [ "## 5. Inspect influential words and phrases\n", "\n", "Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ebf3fde7", "metadata": {}, "outputs": [], "source": [ "feature_names = final_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n", "coefficients = pd.Series(\n", " final_pipeline.named_steps[\"model\"].coef_[0],\n", " index=feature_names,\n", " name=\"coefficient\",\n", ")\n", "\n", "top_spam = coefficients.nlargest(20).reset_index()\n", "top_spam.columns = [\"term\", \"coefficient\"]\n", "top_spam[\"direction\"] = \"spam\"\n", "top_ham = coefficients.nsmallest(20).reset_index()\n", "top_ham.columns = [\"term\", \"coefficient\"]\n", "top_ham[\"direction\"] = \"ham\"\n", "top_terms = pd.concat([top_spam, top_ham], ignore_index=True)\n", "top_terms.head(10)\n" ] }, { "cell_type": "markdown", "id": "779dc4d3", "metadata": {}, "source": [ "## 6. Challenge the detector with new messages\n", "\n", "The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "0c27e573", "metadata": {}, "outputs": [], "source": [ "custom_messages = pd.Series([\n", " \"Can we move the project meeting to 3 pm?\",\n", " \"Congratulations! Claim your free cash prize now\",\n", " \"URGENT: verify your account at the link below\",\n", " \"You are a w1nner, reply FR33 to collect your reward\",\n", " \"Please call me when you arrive at the station\",\n", "], name=\"message\")\n", "\n", "custom_probabilities = final_pipeline.predict_proba(custom_messages)[:, 1]\n", "custom_results = pd.DataFrame({\n", " \"message\": custom_messages,\n", " \"spam_probability\": custom_probabilities,\n", " \"predicted_label\": np.where(custom_probabilities >= DECISION_THRESHOLD, \"spam\", \"ham\"),\n", " \"action\": pd.cut(\n", " custom_probabilities,\n", " bins=[-np.inf, 0.30, 0.70, np.inf],\n", " labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n", " right=False,\n", " ).astype(\"string\"),\n", "})\n", "custom_results\n" ] }, { "cell_type": "markdown", "id": "a27debbf", "metadata": {}, "source": [ "## 7. Export evidence and write the analyst summary\n" ] }, { "cell_type": "code", "execution_count": null, "id": "2d3eb2ed", "metadata": {}, "outputs": [], "source": [ "pd.Series(metrics, name=\"value\").to_csv(OUTPUT_DIR / \"capstone_metrics.csv\", header=True)\n", "prediction_table.to_csv(OUTPUT_DIR / \"capstone_predictions.csv\", index=False)\n", "top_terms.to_csv(OUTPUT_DIR / \"capstone_top_terms.csv\", index=False)\n", "\n", "fig, ax = plt.subplots(figsize=(5, 4))\n", "ConfusionMatrixDisplay(\n", " confusion_matrix=cm,\n", " display_labels=[\"ham\", \"spam\"],\n", ").plot(ax=ax, cmap=\"Blues\", colorbar=False)\n", "ax.set_title(\"SMS Spam Detector — Test Set\")\n", "fig.tight_layout()\n", "fig.savefig(OUTPUT_DIR / \"capstone_confusion_matrix.png\", dpi=160, bbox_inches=\"tight\")\n", "plt.close(fig)\n", "\n", "summary = f'''SMS Spam Detector — Analyst Summary\n", "===================================\n", "Source dataset: UCI SMS Spam Collection\n", "Clean rows: {len(clean_df)}\n", "Training rows: {len(X_train)}\n", "Test rows: {len(X_test)}\n", "Overall spam rate: {y.mean():.2%}\n", "Decision threshold: {DECISION_THRESHOLD:.2f}\n", "\n", "Test performance\n", "----------------\n", "Accuracy: {metrics['accuracy']:.3f}\n", "Spam precision: {metrics['spam_precision']:.3f}\n", "Spam recall: {metrics['spam_recall']:.3f}\n", "Spam F1: {metrics['spam_f1']:.3f}\n", "False positives: {fp}\n", "False negatives: {fn}\n", "\n", "Operational interpretation\n", "--------------------------\n", "Messages below 0.30 are allowed, scores from 0.30 to below 0.70 are sent\n", "for analyst review, and scores of 0.70 or more are quarantined.\n", "\n", "Limitation\n", "----------\n", "This classroom model uses SMS body text only. It does not inspect email headers,\n", "sender domains, attachments, destination URLs, or changing attacker behavior.\n", "'''\n", "summary = textwrap.dedent(summary).strip() + \"\\n\"\n", "(OUTPUT_DIR / \"capstone_summary.txt\").write_text(summary, encoding=\"utf-8\")\n", "\n", "print(summary)\n" ] }, { "cell_type": "markdown", "id": "4ad1b775", "metadata": {}, "source": [ "## Self-check\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3a650060", "metadata": {}, "outputs": [], "source": [ "expected_files = {\n", " \"capstone_metrics.csv\",\n", " \"capstone_predictions.csv\",\n", " \"capstone_top_terms.csv\",\n", " \"capstone_confusion_matrix.png\",\n", " \"capstone_summary.txt\",\n", "}\n", "\n", "assert set(clean_df[\"target\"].unique()) == {0, 1}\n", "assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n", "assert abs(y_train.mean() - y_test.mean()) < 0.02\n", "assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n", "assert len(test_predictions) == len(y_test)\n", "assert cm.shape == (2, 2)\n", "assert (tn + fp + fn + tp) == len(y_test)\n", "assert len(prediction_table) == len(y_test)\n", "assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n", "assert len(top_terms) == 40\n", "assert len(custom_results) == 5\n", "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n", "assert \"Limitation\" in summary\n", "print(\"✓ Lab 6 completed\")\n" ] }, { "cell_type": "markdown", "id": "f680874c", "metadata": {}, "source": [ "## Suggested answers\n", "\n", "1. **What do the false positives and false negatives mean to a security team?** \n", " False positives consume review effort or block legitimate communication. False negatives allow spam through and may expose users to fraud or malicious links. The acceptable balance depends on organizational cost and risk.\n", "2. **Why should the custom obfuscated message be included in the capstone?** \n", " It checks whether the learned word vocabulary is brittle when attackers alter spelling. A failure is useful evidence for considering character n-grams or additional signals later.\n", "3. **What evidence is still missing before this classroom model could support real email security decisions?** \n", " The model needs representative current email data, sender and header features, URL and attachment analysis, temporal validation, calibrated policy costs, adversarial testing, drift monitoring, privacy review, and human oversight.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }