363 lines
12 KiB
Plaintext
363 lines
12 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b438e257",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Module 4 — Lab 5: Probability, Thresholds, and Interpretation\n",
|
||
"\n",
|
||
"\n",
|
||
"**Topics:** Logistic Regression, predict_proba, validation, thresholds, coefficients, actions\n",
|
||
"\n",
|
||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e52f8f6f",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Learning objectives\n",
|
||
"\n",
|
||
"After completing this lab, you will be able to:\n",
|
||
"\n",
|
||
"- reserve separate training, validation, and test partitions;\n",
|
||
"- convert Logistic Regression probabilities into decisions;\n",
|
||
"- choose a threshold using validation data only;\n",
|
||
"- measure the threshold trade-off between false positives and false negatives;\n",
|
||
"- interpret influential terms through model coefficients;\n",
|
||
"- map model scores to allow, analyst-review, and quarantine actions.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "840495c7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from pathlib import Path\n",
|
||
"import numpy as np\n",
|
||
"import pandas as pd\n",
|
||
"\n",
|
||
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||
"from sklearn.linear_model import LogisticRegression\n",
|
||
"from sklearn.metrics import (\n",
|
||
" accuracy_score,\n",
|
||
" confusion_matrix,\n",
|
||
" f1_score,\n",
|
||
" precision_score,\n",
|
||
" recall_score,\n",
|
||
")\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"from sklearn.pipeline import Pipeline\n",
|
||
"\n",
|
||
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
|
||
"RANDOM_STATE = 42\n",
|
||
"THRESHOLDS = np.arange(0.20, 0.81, 0.05)\n",
|
||
"\n",
|
||
"if not DATA_PATH.exists():\n",
|
||
" raise FileNotFoundError(\n",
|
||
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
|
||
" )\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a4aa5a16",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. Prepare train, validation, and test data\n",
|
||
"\n",
|
||
"Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "47bfe88f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
|
||
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
|
||
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
|
||
"df = df.dropna(subset=[\"label\", \"message\"])\n",
|
||
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
|
||
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
|
||
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
|
||
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
|
||
"\n",
|
||
"X = df[\"message\"]\n",
|
||
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cf3ba33b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"X_train, X_temp, y_train, y_temp = train_test_split(\n",
|
||
" X,\n",
|
||
" y,\n",
|
||
" test_size=0.40,\n",
|
||
" random_state=RANDOM_STATE,\n",
|
||
" stratify=y,\n",
|
||
")\n",
|
||
"X_valid, X_test, y_valid, y_test = train_test_split(\n",
|
||
" X_temp,\n",
|
||
" y_temp,\n",
|
||
" test_size=0.50,\n",
|
||
" random_state=RANDOM_STATE,\n",
|
||
" stratify=y_temp,\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"Train / validation / test:\", len(X_train), len(X_valid), len(X_test))\n",
|
||
"print(\"Spam rates:\", f\"{y_train.mean():.2%}\", f\"{y_valid.mean():.2%}\", f\"{y_test.mean():.2%}\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "82a1cadd",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Train a probabilistic text classifier\n",
|
||
"\n",
|
||
"Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ecb39196",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"lr_pipeline = Pipeline([\n",
|
||
" (\"tfidf\", TfidfVectorizer(\n",
|
||
" lowercase=True,\n",
|
||
" ngram_range=(1, 2),\n",
|
||
" min_df=1,\n",
|
||
" sublinear_tf=True,\n",
|
||
" )),\n",
|
||
" (\"model\", LogisticRegression(\n",
|
||
" max_iter=2000,\n",
|
||
" random_state=RANDOM_STATE,\n",
|
||
" )),\n",
|
||
"])\n",
|
||
"lr_pipeline.fit(X_train, y_train)\n",
|
||
"valid_probabilities = lr_pipeline.predict_proba(X_valid)[:, 1]\n",
|
||
"\n",
|
||
"print(\"First probabilities:\", np.round(valid_probabilities[:10], 3))\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b2e41b60",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Select a threshold on validation data\n",
|
||
"\n",
|
||
"The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "54cf81c8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"rows = []\n",
|
||
"for threshold in THRESHOLDS:\n",
|
||
" predictions = (valid_probabilities >= threshold).astype(int)\n",
|
||
" cm = confusion_matrix(y_valid, predictions, labels=[0, 1])\n",
|
||
" tn, fp, fn, tp = cm.ravel()\n",
|
||
" rows.append({\n",
|
||
" \"threshold\": threshold,\n",
|
||
" \"spam_precision\": precision_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
|
||
" \"spam_recall\": recall_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
|
||
" \"spam_f1\": f1_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
|
||
" \"false_positives\": int(fp),\n",
|
||
" \"false_negatives\": int(fn),\n",
|
||
" })\n",
|
||
"\n",
|
||
"threshold_table = pd.DataFrame(rows).set_index(\"threshold\")\n",
|
||
"eligible = threshold_table[threshold_table[\"spam_recall\"].ge(0.80)]\n",
|
||
"if eligible.empty:\n",
|
||
" selected_threshold = float(threshold_table[\"spam_f1\"].idxmax())\n",
|
||
"else:\n",
|
||
" selected_threshold = float(\n",
|
||
" eligible.sort_values([\"spam_precision\", \"spam_f1\"], ascending=False).index[0]\n",
|
||
" )\n",
|
||
"\n",
|
||
"print(threshold_table.round(4))\n",
|
||
"print(\"Selected threshold:\", selected_threshold)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b1658a9c",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. Evaluate once on the untouched test set\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f5d43c65",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"test_probabilities = lr_pipeline.predict_proba(X_test)[:, 1]\n",
|
||
"test_predictions = (test_probabilities >= selected_threshold).astype(int)\n",
|
||
"test_cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n",
|
||
"tn, fp, fn, tp = test_cm.ravel()\n",
|
||
"test_metrics = {\n",
|
||
" \"threshold\": selected_threshold,\n",
|
||
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
|
||
" \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
|
||
" \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
|
||
" \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
|
||
" \"false_positives\": int(fp),\n",
|
||
" \"false_negatives\": int(fn),\n",
|
||
"}\n",
|
||
"\n",
|
||
"print(pd.Series(test_metrics).round(4))\n",
|
||
"print(test_cm)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f41cbbb5",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5. Interpret influential terms\n",
|
||
"\n",
|
||
"Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "87ea0bc0",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"feature_names = lr_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n",
|
||
"coefficients = lr_pipeline.named_steps[\"model\"].coef_[0]\n",
|
||
"coefficient_series = pd.Series(coefficients, index=feature_names)\n",
|
||
"\n",
|
||
"top_ham = coefficient_series.nsmallest(15).rename(\"coefficient\").reset_index()\n",
|
||
"top_ham.columns = [\"term\", \"coefficient\"]\n",
|
||
"top_ham[\"direction\"] = \"ham\"\n",
|
||
"\n",
|
||
"top_spam = coefficient_series.nlargest(15).rename(\"coefficient\").reset_index()\n",
|
||
"top_spam.columns = [\"term\", \"coefficient\"]\n",
|
||
"top_spam[\"direction\"] = \"spam\"\n",
|
||
"\n",
|
||
"top_features = pd.concat([top_spam, top_ham], ignore_index=True)\n",
|
||
"top_features\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7705ca0e",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6. Map probabilities to actions\n",
|
||
"\n",
|
||
"Use fixed demonstration bands: below 0.30 is `allow`, 0.30–0.70 is `analyst_review`, and 0.70 or above is `quarantine`. A production policy would set these boundaries from business cost, capacity, and validation evidence.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9a32d7b4",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"policy_actions = pd.cut(\n",
|
||
" test_probabilities,\n",
|
||
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
|
||
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
|
||
" right=False,\n",
|
||
")\n",
|
||
"action_summary = pd.crosstab(\n",
|
||
" pd.Series(y_test.to_numpy(), name=\"actual\"),\n",
|
||
" pd.Series(policy_actions, name=\"action\"),\n",
|
||
").rename(index={0: \"ham\", 1: \"spam\"})\n",
|
||
"action_summary\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d9e5fb09",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Self-check\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "3da1834c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"assert len(X_train) + len(X_valid) + len(X_test) == len(df)\n",
|
||
"assert set(lr_pipeline.named_steps) == {\"tfidf\", \"model\"}\n",
|
||
"assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))\n",
|
||
"assert threshold_table.index.is_monotonic_increasing\n",
|
||
"assert selected_threshold in threshold_table.index\n",
|
||
"assert len(test_predictions) == len(y_test)\n",
|
||
"assert test_cm.shape == (2, 2)\n",
|
||
"assert len(top_features) == 30\n",
|
||
"assert set(top_features[\"direction\"]) == {\"ham\", \"spam\"}\n",
|
||
"assert set(policy_actions.astype(\"string\").unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
|
||
"print(\"✓ Lab 5 completed\")\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d681ad09",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Suggested answers\n",
|
||
"\n",
|
||
"1. **Why is a validation set needed when selecting a threshold?** \n",
|
||
" Using test labels to choose the threshold would tune the policy to the final evaluation data. Validation data supports selection while keeping the test estimate independent.\n",
|
||
"2. **What usually happens to spam recall when the threshold is lowered?** \n",
|
||
" More messages exceed the lower threshold, so spam recall usually rises, but false positives may also increase and precision may fall.\n",
|
||
"3. **Why are large positive coefficients not proof that a term causes spam?** \n",
|
||
" A coefficient records an association learned from this corpus and representation. It may reflect dataset-specific patterns, confounding, or shortcuts rather than causation.\n"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.6"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|