{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4c3ac6bc",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
    "\n",
    "\n",
    "**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb25bc5b",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- compare several classifiers on one fixed train-test split;\n",
    "- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
    "- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
    "- connect false positives and false negatives to operational consequences;\n",
    "- explain why kernel SVM is not required for this classroom workflow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "603330c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import LogisticRegression, Perceptron\n",
    "from sklearn.metrics import (\n",
    "    accuracy_score,\n",
    "    confusion_matrix,\n",
    "    f1_score,\n",
    "    precision_score,\n",
    "    recall_score,\n",
    ")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae34196",
   "metadata": {},
   "source": [
    "## 1. Prepare one fixed experiment\n",
    "\n",
    "Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1f43dbfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
    "df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
    "df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
    "df = df.dropna(subset=[\"label\", \"message\"])\n",
    "df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
    "conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
    "df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
    "df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
    "\n",
    "X = df[\"message\"]\n",
    "y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e23c9db",
   "metadata": {},
   "source": [
    "## 2. Define the candidate pipelines\n",
    "\n",
    "All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5af7ba3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Models: None\n"
     ]
    }
   ],
   "source": [
    "# TODO 1: create four pipelines with identical TF-IDF settings.\n",
    "# Models: MultinomialNB, Perceptron, LinearSVC, and LogisticRegression.\n",
    "models = None\n",
    "\n",
    "print(\"Models:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86aa5371",
   "metadata": {},
   "source": [
    "## 3. Train and evaluate fairly\n",
    "\n",
    "Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dd93163a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 2: fit every pipeline on the same training split.\n",
    "# TODO 3: record accuracy, spam precision, spam recall, spam F1,\n",
    "# false positives, and false negatives for each model.\n",
    "fitted_models = None\n",
    "predictions = None\n",
    "comparison = None\n",
    "\n",
    "comparison\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b281735",
   "metadata": {},
   "source": [
    "## 4. Select and inspect the strongest model\n",
    "\n",
    "For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b327c2c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model by spam F1: None\n",
      "Number of errors: None\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'head'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[5], line 11\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mBest model by spam F1:\u001b[39m\u001b[38;5;124m\"\u001b[39m, best_model_name)\n\u001b[1;32m     10\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNumber of errors:\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m---> 11\u001b[0m \u001b[43mbest_errors\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mhead\u001b[49m(\u001b[38;5;241m10\u001b[39m)\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'head'"
     ]
    }
   ],
   "source": [
    "# TODO 4: select the model with the highest spam F1.\n",
    "best_model_name = None\n",
    "best_pipeline = None\n",
    "best_predictions = None\n",
    "\n",
    "# TODO 5: create a DataFrame containing the best model's test errors.\n",
    "best_errors = None\n",
    "\n",
    "print(\"Best model by spam F1:\", best_model_name)\n",
    "print(\"Number of errors:\", None)\n",
    "best_errors.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2eb368a",
   "metadata": {},
   "source": [
    "## Interpretation checkpoint\n",
    "\n",
    "A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39f0a30",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4666cc68",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
    "assert set(fitted_models) == set(models)\n",
    "assert set(predictions) == set(models)\n",
    "assert comparison.shape == (4, 6)\n",
    "assert comparison.index.is_unique\n",
    "assert comparison[\"spam_f1\"].between(0, 1).all()\n",
    "assert best_model_name in models\n",
    "assert len(best_predictions) == len(y_test)\n",
    "assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
    "print(\"✓ Lab 4 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e7f451c",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why must every model use the same data split?\n",
    "2. Why is a linear SVM appropriate for TF-IDF features?\n",
    "3. Why might the highest-accuracy model still be unsuitable for spam filtering?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}