295 lines
9.4 KiB
Plaintext
295 lines
9.4 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "14ac1890",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline\n",
|
|
"\n",
|
|
"**Topics:** Pipeline, MultinomialNB, confusion matrix, baseline, error analysis\n",
|
|
"\n",
|
|
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "af2dc7be",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Learning objectives\n",
|
|
"\n",
|
|
"After completing this lab, you will be able to:\n",
|
|
"\n",
|
|
"- place TF-IDF and Multinomial Naive Bayes in one pipeline;\n",
|
|
"- explain how the pipeline prevents preprocessing leakage;\n",
|
|
"- calculate spam precision, recall, and F1;\n",
|
|
"- compare a trained classifier with a naive baseline;\n",
|
|
"- inspect false positives and false negatives.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bced8aa9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pathlib import Path\n",
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"from sklearn.dummy import DummyClassifier\n",
|
|
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
|
"from sklearn.metrics import (\n",
|
|
" accuracy_score,\n",
|
|
" classification_report,\n",
|
|
" confusion_matrix,\n",
|
|
" f1_score,\n",
|
|
" precision_score,\n",
|
|
" recall_score,\n",
|
|
")\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.naive_bayes import MultinomialNB\n",
|
|
"from sklearn.pipeline import Pipeline\n",
|
|
"\n",
|
|
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
|
|
"RANDOM_STATE = 42\n",
|
|
"\n",
|
|
"if not DATA_PATH.exists():\n",
|
|
" raise FileNotFoundError(\n",
|
|
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aa622049",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Load, audit, and split the messages\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9efd4a80",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
|
|
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
|
|
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
|
|
"df = df.dropna(subset=[\"label\", \"message\"])\n",
|
|
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
|
|
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
|
|
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
|
|
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
|
|
"\n",
|
|
"X = df[\"message\"]\n",
|
|
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
|
|
"\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
|
|
")\n",
|
|
"print(\"Training messages:\", len(X_train))\n",
|
|
"print(\"Test messages:\", len(X_test))\n",
|
|
"print(\"Training spam rate:\", f\"{y_train.mean():.2%}\")\n",
|
|
"print(\"Test spam rate:\", f\"{y_test.mean():.2%}\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "263fa9cf",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Build and train one pipeline\n",
|
|
"\n",
|
|
"The pipeline fits TF-IDF only on `X_train`. During prediction, it applies the already-fitted transformation to `X_test`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aed27122",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"nb_pipeline = Pipeline([\n",
|
|
" (\"tfidf\", TfidfVectorizer(lowercase=True, ngram_range=(1, 1))),\n",
|
|
" (\"model\", MultinomialNB(alpha=1.0)),\n",
|
|
"])\n",
|
|
"nb_pipeline.fit(X_train, y_train)\n",
|
|
"y_pred = nb_pipeline.predict(X_test)\n",
|
|
"\n",
|
|
"print(nb_pipeline)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "50d5bc04",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Evaluate the spam class\n",
|
|
"\n",
|
|
"- **Spam precision:** when the model blocks a message, how often is it truly spam?\n",
|
|
"- **Spam recall:** of all spam messages, how many did the model catch?\n",
|
|
"- **Spam F1:** harmonic mean of spam precision and recall.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "32b32feb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
|
|
"tn, fp, fn, tp = cm.ravel()\n",
|
|
"\n",
|
|
"nb_metrics = {\n",
|
|
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
|
|
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
"}\n",
|
|
"\n",
|
|
"print(pd.Series(nb_metrics).round(4))\n",
|
|
"print(\"\\nClassification report:\")\n",
|
|
"print(classification_report(y_test, y_pred, target_names=[\"ham\", \"spam\"], zero_division=0))\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a7568ba",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Compare with a naive baseline\n",
|
|
"\n",
|
|
"Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7198e0cf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dummy = DummyClassifier(strategy=\"most_frequent\")\n",
|
|
"dummy.fit(X_train.to_frame(), y_train)\n",
|
|
"dummy_pred = dummy.predict(X_test.to_frame())\n",
|
|
"\n",
|
|
"comparison = pd.DataFrame([\n",
|
|
" {\n",
|
|
" \"model\": \"Most-frequent baseline\",\n",
|
|
" \"accuracy\": accuracy_score(y_test, dummy_pred),\n",
|
|
" \"spam_f1\": f1_score(y_test, dummy_pred, pos_label=1, zero_division=0),\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"model\": \"TF-IDF + MultinomialNB\",\n",
|
|
" \"accuracy\": nb_metrics[\"accuracy\"],\n",
|
|
" \"spam_f1\": nb_metrics[\"spam_f1\"],\n",
|
|
" },\n",
|
|
"]).set_index(\"model\")\n",
|
|
"comparison\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "56233ccf",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 5. Inspect operational errors\n",
|
|
"\n",
|
|
"A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e601a648",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"error_analysis = pd.DataFrame({\n",
|
|
" \"message\": X_test.to_numpy(),\n",
|
|
" \"actual\": y_test.to_numpy(),\n",
|
|
" \"predicted\": y_pred,\n",
|
|
"})\n",
|
|
"errors = error_analysis[error_analysis[\"actual\"].ne(error_analysis[\"predicted\"])].copy()\n",
|
|
"errors[\"error_type\"] = np.where(\n",
|
|
" (errors[\"actual\"] == 0) & (errors[\"predicted\"] == 1),\n",
|
|
" \"false_positive\",\n",
|
|
" \"false_negative\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(errors[\"error_type\"].value_counts())\n",
|
|
"errors.head(10)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a818ba0b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Self-check\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "72c65672",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"assert list(nb_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
|
|
"assert isinstance(nb_pipeline.named_steps[\"model\"], MultinomialNB)\n",
|
|
"assert len(y_pred) == len(y_test)\n",
|
|
"assert cm.shape == (2, 2)\n",
|
|
"assert (tn + fp + fn + tp) == len(y_test)\n",
|
|
"assert set(nb_metrics) == {\"accuracy\", \"spam_precision\", \"spam_recall\", \"spam_f1\"}\n",
|
|
"assert comparison.loc[\"Most-frequent baseline\", \"spam_f1\"] == 0\n",
|
|
"assert set(errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
|
|
"print(\"✓ Lab 3 completed\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7f0d0753",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Suggested answers\n",
|
|
"\n",
|
|
"1. **Why is the most-frequent classifier an important baseline?** \n",
|
|
" It reveals how much accuracy comes from class imbalance alone. A useful spam detector must outperform the baseline on spam-sensitive metrics.\n",
|
|
"2. **Which operational harm is represented by a false positive?** \n",
|
|
" A legitimate message is incorrectly blocked or quarantined, which can interrupt communication and reduce user trust.\n",
|
|
"3. **Why should TF-IDF and the model be placed in one pipeline?** \n",
|
|
" The pipeline guarantees that vocabulary and IDF statistics are learned only during fitting and then reused consistently, reducing leakage and preprocessing mistakes.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|