{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64dbadae",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 5: Probability, Thresholds, and Interpretation\n",
    "\n",
    "  \n",
    "**Topics:** Logistic Regression, predict_proba, validation, thresholds, coefficients, actions\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4da9f59",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- reserve separate training, validation, and test partitions;\n",
    "- convert Logistic Regression probabilities into decisions;\n",
    "- choose a threshold using validation data only;\n",
    "- measure the threshold trade-off between false positives and false negatives;\n",
    "- interpret influential terms through model coefficients;\n",
    "- map model scores to allow, analyst-review, and quarantine actions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9579de3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import (\n",
    "    accuracy_score,\n",
    "    confusion_matrix,\n",
    "    f1_score,\n",
    "    precision_score,\n",
    "    recall_score,\n",
    ")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "RANDOM_STATE = 42\n",
    "THRESHOLDS = np.arange(0.20, 0.81, 0.05)\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec160895",
   "metadata": {},
   "source": [
    "## 1. Prepare train, validation, and test data\n",
    "\n",
    "Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5880993f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
    "df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
    "df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
    "df = df.dropna(subset=[\"label\", \"message\"])\n",
    "df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
    "conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
    "df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
    "df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
    "\n",
    "X = df[\"message\"]\n",
    "y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9d2bae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: split the data into 60% training and 40% temporary data.\n",
    "X_train, X_temp, y_train, y_temp = None, None, None, None\n",
    "\n",
    "# TODO 2: split the temporary data equally into validation and test sets.\n",
    "X_valid, X_test, y_valid, y_test = None, None, None, None\n",
    "\n",
    "print(\"Train / validation / test:\", None)\n",
    "print(\"Spam rates:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91f38f11",
   "metadata": {},
   "source": [
    "## 2. Train a probabilistic text classifier\n",
    "\n",
    "Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbd38eaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: create and fit a TF-IDF + Logistic Regression pipeline.\n",
    "lr_pipeline = None\n",
    "\n",
    "# TODO 4: calculate validation spam probabilities.\n",
    "valid_probabilities = None\n",
    "\n",
    "print(\"First probabilities:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdd0579e",
   "metadata": {},
   "source": [
    "## 3. Select a threshold on validation data\n",
    "\n",
    "The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "042a6426",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: evaluate every threshold in THRESHOLDS on validation data.\n",
    "threshold_table = None\n",
    "\n",
    "# TODO 6: among thresholds with spam recall >= 0.80, select the one\n",
    "# with the highest precision. If none qualify, use the highest F1.\n",
    "selected_threshold = None\n",
    "\n",
    "print(threshold_table.round(4))\n",
    "print(\"Selected threshold:\", selected_threshold)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f33c692",
   "metadata": {},
   "source": [
    "## 4. Evaluate once on the untouched test set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7567738",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: evaluate the selected threshold once on the untouched test set.\n",
    "test_probabilities = None\n",
    "test_predictions = None\n",
    "test_cm = None\n",
    "test_metrics = None\n",
    "\n",
    "print(pd.Series(test_metrics).round(4))\n",
    "print(test_cm)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffd1f1c",
   "metadata": {},
   "source": [
    "## 5. Interpret influential terms\n",
    "\n",
    "Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8996ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 8: extract the fitted vocabulary and Logistic Regression coefficients.\n",
    "feature_names = None\n",
    "coefficients = None\n",
    "\n",
    "# TODO 9: create a table with the 15 strongest ham terms and\n",
    "# the 15 strongest spam terms.\n",
    "top_features = None\n",
    "top_features\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42879342",
   "metadata": {},
   "source": [
    "## 6. Map probabilities to actions\n",
    "\n",
    "Use fixed demonstration bands: below 0.30 is `allow`, 0.30–0.70 is `analyst_review`, and 0.70 or above is `quarantine`. A production policy would set these boundaries from business cost, capacity, and validation evidence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7031034a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 10: map test probabilities to allow, analyst_review, or quarantine.\n",
    "policy_actions = None\n",
    "action_summary = None\n",
    "\n",
    "action_summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3545892",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f99dce7",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(X_train) + len(X_valid) + len(X_test) == len(df)\n",
    "assert set(lr_pipeline.named_steps) == {\"tfidf\", \"model\"}\n",
    "assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))\n",
    "assert threshold_table.index.is_monotonic_increasing\n",
    "assert selected_threshold in threshold_table.index\n",
    "assert len(test_predictions) == len(y_test)\n",
    "assert test_cm.shape == (2, 2)\n",
    "assert len(top_features) == 30\n",
    "assert set(top_features[\"direction\"]) == {\"ham\", \"spam\"}\n",
    "assert set(policy_actions.astype(\"string\").unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
    "print(\"✓ Lab 5 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913d3185",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why is a validation set needed when selecting a threshold?\n",
    "2. What usually happens to spam recall when the threshold is lowered?\n",
    "3. Why are large positive coefficients not proof that a term causes spam?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}