280 lines
9.0 KiB
Plaintext
280 lines
9.0 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4c3ac6bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
|
|
"\n",
|
|
"\n",
|
|
"**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
|
|
"\n",
|
|
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fb25bc5b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Learning objectives\n",
|
|
"\n",
|
|
"After completing this lab, you will be able to:\n",
|
|
"\n",
|
|
"- compare several classifiers on one fixed train-test split;\n",
|
|
"- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
|
|
"- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
|
|
"- connect false positives and false negatives to operational consequences;\n",
|
|
"- explain why kernel SVM is not required for this classroom workflow.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "603330c3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pathlib import Path\n",
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
|
"from sklearn.linear_model import LogisticRegression, Perceptron\n",
|
|
"from sklearn.metrics import (\n",
|
|
" accuracy_score,\n",
|
|
" confusion_matrix,\n",
|
|
" f1_score,\n",
|
|
" precision_score,\n",
|
|
" recall_score,\n",
|
|
")\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.naive_bayes import MultinomialNB\n",
|
|
"from sklearn.pipeline import Pipeline\n",
|
|
"from sklearn.svm import LinearSVC\n",
|
|
"\n",
|
|
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
|
|
"RANDOM_STATE = 42\n",
|
|
"\n",
|
|
"if not DATA_PATH.exists():\n",
|
|
" raise FileNotFoundError(\n",
|
|
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cae34196",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Prepare one fixed experiment\n",
|
|
"\n",
|
|
"Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "1f43dbfe",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
|
|
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
|
|
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
|
|
"df = df.dropna(subset=[\"label\", \"message\"])\n",
|
|
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
|
|
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
|
|
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
|
|
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
|
|
"\n",
|
|
"X = df[\"message\"]\n",
|
|
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
|
|
")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2e23c9db",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Define the candidate pipelines\n",
|
|
"\n",
|
|
"All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "5af7ba3e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Models: None\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# TODO 1: create four pipelines with identical TF-IDF settings.\n",
|
|
"# Models: MultinomialNB, Perceptron, LinearSVC, and LogisticRegression.\n",
|
|
"models = None\n",
|
|
"\n",
|
|
"print(\"Models:\", None)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86aa5371",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Train and evaluate fairly\n",
|
|
"\n",
|
|
"Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "dd93163a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 2: fit every pipeline on the same training split.\n",
|
|
"# TODO 3: record accuracy, spam precision, spam recall, spam F1,\n",
|
|
"# false positives, and false negatives for each model.\n",
|
|
"fitted_models = None\n",
|
|
"predictions = None\n",
|
|
"comparison = None\n",
|
|
"\n",
|
|
"comparison\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1b281735",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Select and inspect the strongest model\n",
|
|
"\n",
|
|
"For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "b327c2c6",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Best model by spam F1: None\n",
|
|
"Number of errors: None\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "AttributeError",
|
|
"evalue": "'NoneType' object has no attribute 'head'",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[5], line 11\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mBest model by spam F1:\u001b[39m\u001b[38;5;124m\"\u001b[39m, best_model_name)\n\u001b[1;32m 10\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNumber of errors:\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m---> 11\u001b[0m \u001b[43mbest_errors\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mhead\u001b[49m(\u001b[38;5;241m10\u001b[39m)\n",
|
|
"\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'head'"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# TODO 4: select the model with the highest spam F1.\n",
|
|
"best_model_name = None\n",
|
|
"best_pipeline = None\n",
|
|
"best_predictions = None\n",
|
|
"\n",
|
|
"# TODO 5: create a DataFrame containing the best model's test errors.\n",
|
|
"best_errors = None\n",
|
|
"\n",
|
|
"print(\"Best model by spam F1:\", best_model_name)\n",
|
|
"print(\"Number of errors:\", None)\n",
|
|
"best_errors.head(10)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2eb368a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Interpretation checkpoint\n",
|
|
"\n",
|
|
"A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e39f0a30",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Self-check\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4666cc68",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
|
|
"assert set(fitted_models) == set(models)\n",
|
|
"assert set(predictions) == set(models)\n",
|
|
"assert comparison.shape == (4, 6)\n",
|
|
"assert comparison.index.is_unique\n",
|
|
"assert comparison[\"spam_f1\"].between(0, 1).all()\n",
|
|
"assert best_model_name in models\n",
|
|
"assert len(best_predictions) == len(y_test)\n",
|
|
"assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
|
|
"print(\"✓ Lab 4 completed\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e7f451c",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Reflection questions\n",
|
|
"\n",
|
|
"1. Why must every model use the same data split?\n",
|
|
"2. Why is a linear SVM appropriate for TF-IDF features?\n",
|
|
"3. Why might the highest-accuracy model still be unsuitable for spam filtering?\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|