CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
@@ -0,0 +1,354 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "59b46ce9",
+   "metadata": {},
+   "source": [
+    "# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n",
+    "\n",
+    "\n",
+    "**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n",
+    "\n",
+    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "06204436",
+   "metadata": {},
+   "source": [
+    "## Mission\n",
+    "\n",
+    "Build one complete, reproducible, and leakage-safe spam-detection workflow:\n",
+    "\n",
+    "1. load and audit the UCI SMS Spam Collection;\n",
+    "2. create one stratified holdout test set;\n",
+    "3. train a word-level TF-IDF and Logistic Regression pipeline;\n",
+    "4. evaluate spam-sensitive metrics and operational errors;\n",
+    "5. inspect influential terms and test custom messages;\n",
+    "6. export evidence and an analyst-facing summary.\n",
+    "\n",
+    "The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8afc13cc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import textwrap\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "from sklearn.metrics import (\n",
+    "    ConfusionMatrixDisplay,\n",
+    "    accuracy_score,\n",
+    "    classification_report,\n",
+    "    confusion_matrix,\n",
+    "    f1_score,\n",
+    "    precision_score,\n",
+    "    recall_score,\n",
+    ")\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.pipeline import Pipeline\n",
+    "\n",
+    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
+    "OUTPUT_DIR = Path(\"output\")\n",
+    "RANDOM_STATE = 42\n",
+    "DECISION_THRESHOLD = 0.50\n",
+    "\n",
+    "if not DATA_PATH.exists():\n",
+    "    raise FileNotFoundError(\n",
+    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
+    "    )\n",
+    "\n",
+    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0bf5bfb8",
+   "metadata": {},
+   "source": [
+    "## 1. Load, audit, and split the data\n",
+    "\n",
+    "Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5092a6b4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 1: load the UCI file, normalize the two columns, remove invalid\n",
+    "# rows and conflicts, remove exact duplicates, and create target.\n",
+    "raw_df = None\n",
+    "clean_df = None\n",
+    "\n",
+    "# TODO 2: create X and y, then make a stratified 80/20 split.\n",
+    "X = None\n",
+    "y = None\n",
+    "X_train, X_test, y_train, y_test = None, None, None, None\n",
+    "\n",
+    "print(\"Raw rows:\", None)\n",
+    "print(\"Clean rows:\", None)\n",
+    "print(\"Training / test:\", None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "649bf063",
+   "metadata": {},
+   "source": [
+    "## 2. Build and train the final pipeline\n",
+    "\n",
+    "The pipeline learns its vocabulary and IDF weights from the training messages only.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78a6493b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 3: create a word unigram+bigram TF-IDF and Logistic Regression pipeline.\n",
+    "final_pipeline = None\n",
+    "\n",
+    "# TODO 4: fit on training data and calculate test probabilities and predictions.\n",
+    "test_probabilities = None\n",
+    "test_predictions = None\n",
+    "\n",
+    "print(\"Predictions created:\", None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b99770cb",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate the held-out test set\n",
+    "\n",
+    "Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9ed40674",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 5: calculate the confusion matrix and capstone metrics.\n",
+    "cm = None\n",
+    "tn, fp, fn, tp = None, None, None, None\n",
+    "metrics = None\n",
+    "\n",
+    "print(pd.Series(metrics).round(4))\n",
+    "print(None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3131b3f0",
+   "metadata": {},
+   "source": [
+    "## 4. Create an analyst error table\n",
+    "\n",
+    "Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "95449797",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 6: create prediction_table with message, actual_label,\n",
+    "# predicted_label, spam_probability, action, and error_type.\n",
+    "prediction_table = None\n",
+    "\n",
+    "# TODO 7: keep only false positives and false negatives in error_table.\n",
+    "error_table = None\n",
+    "\n",
+    "print(\"Errors by type:\")\n",
+    "print(None)\n",
+    "error_table.head(10)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e075f283",
+   "metadata": {},
+   "source": [
+    "## 5. Inspect influential words and phrases\n",
+    "\n",
+    "Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8da0477",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 8: extract the 20 strongest spam terms and 20 strongest ham terms.\n",
+    "top_terms = None\n",
+    "top_terms.head(10)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2ecdeca9",
+   "metadata": {},
+   "source": [
+    "## 6. Challenge the detector with new messages\n",
+    "\n",
+    "The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dcb531ea",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "custom_messages = pd.Series([\n",
+    "    \"Can we move the project meeting to 3 pm?\",\n",
+    "    \"Congratulations! Claim your free cash prize now\",\n",
+    "    \"URGENT: verify your account at the link below\",\n",
+    "    \"You are a w1nner, reply FR33 to collect your reward\",\n",
+    "    \"Please call me when you arrive at the station\",\n",
+    "], name=\"message\")\n",
+    "\n",
+    "# TODO 9: score the custom messages and assign the three policy actions.\n",
+    "custom_probabilities = None\n",
+    "custom_results = None\n",
+    "custom_results\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "99e5489b",
+   "metadata": {},
+   "source": [
+    "## 7. Export evidence and write the analyst summary\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f0d90df9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 10: save metrics, predictions, top terms, and a confusion-matrix chart.\n",
+    "# TODO 11: write an analyst summary containing the dataset size, spam rate,\n",
+    "# threshold, four primary metrics, FP/FN counts, and one limitation.\n",
+    "summary = None\n",
+    "\n",
+    "print(summary)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b8fac56b",
+   "metadata": {},
+   "source": [
+    "## Self-check\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1694158b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "expected_files = {\n",
+    "    \"capstone_metrics.csv\",\n",
+    "    \"capstone_predictions.csv\",\n",
+    "    \"capstone_top_terms.csv\",\n",
+    "    \"capstone_confusion_matrix.png\",\n",
+    "    \"capstone_summary.txt\",\n",
+    "}\n",
+    "\n",
+    "assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
+    "assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
+    "assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
+    "assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
+    "assert len(test_predictions) == len(y_test)\n",
+    "assert cm.shape == (2, 2)\n",
+    "assert (tn + fp + fn + tp) == len(y_test)\n",
+    "assert len(prediction_table) == len(y_test)\n",
+    "assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
+    "assert len(top_terms) == 40\n",
+    "assert len(custom_results) == 5\n",
+    "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
+    "assert \"Limitation\" in summary\n",
+    "print(\"✓ Lab 6 completed\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14123800",
+   "metadata": {},
+   "source": [
+    "## Reflection questions\n",
+    "\n",
+    "1. What do the false positives and false negatives mean to a security team?\n",
+    "2. Why should the custom obfuscated message be included in the capstone?\n",
+    "3. What evidence is still missing before this classroom model could support real email security decisions?\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "60c6f317-85da-455b-80bb-b9a2d6f8b556",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e4043e40-4d92-48a4-b5f1-51fe55f2f75b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}