{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac0bd1d0",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline\n",
    "\n",
    "\n",
    "**Topics:** Pipeline, MultinomialNB, confusion matrix, baseline, error analysis\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0a933d7",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- place TF-IDF and Multinomial Naive Bayes in one pipeline;\n",
    "- explain how the pipeline prevents preprocessing leakage;\n",
    "- calculate spam precision, recall, and F1;\n",
    "- compare a trained classifier with a naive baseline;\n",
    "- inspect false positives and false negatives.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43e5d008",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics import (\n",
    "    accuracy_score,\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    f1_score,\n",
    "    precision_score,\n",
    "    recall_score,\n",
    ")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "831471be",
   "metadata": {},
   "source": [
    "## 1. Load, audit, and split the messages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32983efc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
    "df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
    "df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
    "df = df.dropna(subset=[\"label\", \"message\"])\n",
    "df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
    "conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
    "df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
    "df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
    "\n",
    "X = df[\"message\"]\n",
    "y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
    ")\n",
    "print(\"Training messages:\", len(X_train))\n",
    "print(\"Test messages:\", len(X_test))\n",
    "print(\"Training spam rate:\", f\"{y_train.mean():.2%}\")\n",
    "print(\"Test spam rate:\", f\"{y_test.mean():.2%}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd01144",
   "metadata": {},
   "source": [
    "## 2. Build and train one pipeline\n",
    "\n",
    "The pipeline fits TF-IDF only on `X_train`. During prediction, it applies the already-fitted transformation to `X_test`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "458119d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: create a Pipeline named nb_pipeline with a TF-IDF vectorizer\n",
    "# followed by MultinomialNB(alpha=1.0).\n",
    "nb_pipeline = None\n",
    "\n",
    "# TODO 2: fit the pipeline on X_train and y_train, then predict X_test.\n",
    "y_pred = None\n",
    "\n",
    "print(nb_pipeline)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aedf1e8",
   "metadata": {},
   "source": [
    "## 3. Evaluate the spam class\n",
    "\n",
    "- **Spam precision:** when the model blocks a message, how often is it truly spam?\n",
    "- **Spam recall:** of all spam messages, how many did the model catch?\n",
    "- **Spam F1:** harmonic mean of spam precision and recall.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "315b9a3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: calculate the confusion matrix and unpack tn, fp, fn, tp.\n",
    "cm = None\n",
    "tn, fp, fn, tp = None, None, None, None\n",
    "\n",
    "# TODO 4: calculate accuracy and spam precision, recall, and F1.\n",
    "nb_metrics = None\n",
    "\n",
    "print(pd.Series(nb_metrics).round(4))\n",
    "print(\"\\nClassification report:\")\n",
    "print(None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0238d75f",
   "metadata": {},
   "source": [
    "## 4. Compare with a naive baseline\n",
    "\n",
    "Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac5b05ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: train a most_frequent DummyClassifier and predict X_test.\n",
    "dummy = None\n",
    "dummy_pred = None\n",
    "\n",
    "# TODO 6: compare the two models using spam F1 and accuracy.\n",
    "comparison = None\n",
    "comparison\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "198f66ac",
   "metadata": {},
   "source": [
    "## 5. Inspect operational errors\n",
    "\n",
    "A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ae6e63f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: create a DataFrame with message, actual, and predicted columns.\n",
    "error_analysis = None\n",
    "\n",
    "# TODO 8: keep only incorrect predictions and add an error_type column\n",
    "# containing false_positive or false_negative.\n",
    "errors = None\n",
    "\n",
    "print(errors[\"error_type\"].value_counts())\n",
    "errors.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b3715ec",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1962e1aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert list(nb_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
    "assert isinstance(nb_pipeline.named_steps[\"model\"], MultinomialNB)\n",
    "assert len(y_pred) == len(y_test)\n",
    "assert cm.shape == (2, 2)\n",
    "assert (tn + fp + fn + tp) == len(y_test)\n",
    "assert set(nb_metrics) == {\"accuracy\", \"spam_precision\", \"spam_recall\", \"spam_f1\"}\n",
    "assert comparison.loc[\"Most-frequent baseline\", \"spam_f1\"] == 0\n",
    "assert set(errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
    "print(\"✓ Lab 3 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1ea84b",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why is the most-frequent classifier an important baseline?\n",
    "2. Which operational harm is represented by a false positive?\n",
    "3. Why should TF-IDF and the model be placed in one pipeline?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}