{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59b46ce9",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n",
    "\n",
    "\n",
    "**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06204436",
   "metadata": {},
   "source": [
    "## Mission\n",
    "\n",
    "Build one complete, reproducible, and leakage-safe spam-detection workflow:\n",
    "\n",
    "1. load and audit the UCI SMS Spam Collection;\n",
    "2. create one stratified holdout test set;\n",
    "3. train a word-level TF-IDF and Logistic Regression pipeline;\n",
    "4. evaluate spam-sensitive metrics and operational errors;\n",
    "5. inspect influential terms and test custom messages;\n",
    "6. export evidence and an analyst-facing summary.\n",
    "\n",
    "The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8afc13cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import textwrap\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import (\n",
    "    ConfusionMatrixDisplay,\n",
    "    accuracy_score,\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    f1_score,\n",
    "    precision_score,\n",
    "    recall_score,\n",
    ")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "RANDOM_STATE = 42\n",
    "DECISION_THRESHOLD = 0.50\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
    "    )\n",
    "\n",
    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf5bfb8",
   "metadata": {},
   "source": [
    "## 1. Load, audit, and split the data\n",
    "\n",
    "Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5092a6b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load the UCI file, normalize the two columns, remove invalid\n",
    "# rows and conflicts, remove exact duplicates, and create target.\n",
    "raw_df = None\n",
    "clean_df = None\n",
    "\n",
    "# TODO 2: create X and y, then make a stratified 80/20 split.\n",
    "X = None\n",
    "y = None\n",
    "X_train, X_test, y_train, y_test = None, None, None, None\n",
    "\n",
    "print(\"Raw rows:\", None)\n",
    "print(\"Clean rows:\", None)\n",
    "print(\"Training / test:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "649bf063",
   "metadata": {},
   "source": [
    "## 2. Build and train the final pipeline\n",
    "\n",
    "The pipeline learns its vocabulary and IDF weights from the training messages only.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78a6493b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: create a word unigram+bigram TF-IDF and Logistic Regression pipeline.\n",
    "final_pipeline = None\n",
    "\n",
    "# TODO 4: fit on training data and calculate test probabilities and predictions.\n",
    "test_probabilities = None\n",
    "test_predictions = None\n",
    "\n",
    "print(\"Predictions created:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99770cb",
   "metadata": {},
   "source": [
    "## 3. Evaluate the held-out test set\n",
    "\n",
    "Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ed40674",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: calculate the confusion matrix and capstone metrics.\n",
    "cm = None\n",
    "tn, fp, fn, tp = None, None, None, None\n",
    "metrics = None\n",
    "\n",
    "print(pd.Series(metrics).round(4))\n",
    "print(None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3131b3f0",
   "metadata": {},
   "source": [
    "## 4. Create an analyst error table\n",
    "\n",
    "Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95449797",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 6: create prediction_table with message, actual_label,\n",
    "# predicted_label, spam_probability, action, and error_type.\n",
    "prediction_table = None\n",
    "\n",
    "# TODO 7: keep only false positives and false negatives in error_table.\n",
    "error_table = None\n",
    "\n",
    "print(\"Errors by type:\")\n",
    "print(None)\n",
    "error_table.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e075f283",
   "metadata": {},
   "source": [
    "## 5. Inspect influential words and phrases\n",
    "\n",
    "Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8da0477",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 8: extract the 20 strongest spam terms and 20 strongest ham terms.\n",
    "top_terms = None\n",
    "top_terms.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ecdeca9",
   "metadata": {},
   "source": [
    "## 6. Challenge the detector with new messages\n",
    "\n",
    "The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcb531ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_messages = pd.Series([\n",
    "    \"Can we move the project meeting to 3 pm?\",\n",
    "    \"Congratulations! Claim your free cash prize now\",\n",
    "    \"URGENT: verify your account at the link below\",\n",
    "    \"You are a w1nner, reply FR33 to collect your reward\",\n",
    "    \"Please call me when you arrive at the station\",\n",
    "], name=\"message\")\n",
    "\n",
    "# TODO 9: score the custom messages and assign the three policy actions.\n",
    "custom_probabilities = None\n",
    "custom_results = None\n",
    "custom_results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e5489b",
   "metadata": {},
   "source": [
    "## 7. Export evidence and write the analyst summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0d90df9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 10: save metrics, predictions, top terms, and a confusion-matrix chart.\n",
    "# TODO 11: write an analyst summary containing the dataset size, spam rate,\n",
    "# threshold, four primary metrics, FP/FN counts, and one limitation.\n",
    "summary = None\n",
    "\n",
    "print(summary)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8fac56b",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1694158b",
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_files = {\n",
    "    \"capstone_metrics.csv\",\n",
    "    \"capstone_predictions.csv\",\n",
    "    \"capstone_top_terms.csv\",\n",
    "    \"capstone_confusion_matrix.png\",\n",
    "    \"capstone_summary.txt\",\n",
    "}\n",
    "\n",
    "assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
    "assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
    "assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
    "assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
    "assert len(test_predictions) == len(y_test)\n",
    "assert cm.shape == (2, 2)\n",
    "assert (tn + fp + fn + tp) == len(y_test)\n",
    "assert len(prediction_table) == len(y_test)\n",
    "assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
    "assert len(top_terms) == 40\n",
    "assert len(custom_results) == 5\n",
    "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
    "assert \"Limitation\" in summary\n",
    "print(\"✓ Lab 6 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14123800",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. What do the false positives and false negatives mean to a security team?\n",
    "2. Why should the custom obfuscated message be included in the capstone?\n",
    "3. What evidence is still missing before this classroom model could support real email security decisions?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60c6f317-85da-455b-80bb-b9a2d6f8b556",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4043e40-4d92-48a4-b5f1-51fe55f2f75b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}