305 lines
10 KiB
Plaintext
305 lines
10 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d43eebb6",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
|
|
"\n",
|
|
"\n",
|
|
"**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
|
|
"\n",
|
|
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "78892370",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Learning objectives\n",
|
|
"\n",
|
|
"After completing this lab, you will be able to:\n",
|
|
"\n",
|
|
"- compare several classifiers on one fixed train-test split;\n",
|
|
"- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
|
|
"- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
|
|
"- connect false positives and false negatives to operational consequences;\n",
|
|
"- explain why kernel SVM is not required for this classroom workflow.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d9e93987",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pathlib import Path\n",
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
|
"from sklearn.linear_model import LogisticRegression, Perceptron\n",
|
|
"from sklearn.metrics import (\n",
|
|
" accuracy_score,\n",
|
|
" confusion_matrix,\n",
|
|
" f1_score,\n",
|
|
" precision_score,\n",
|
|
" recall_score,\n",
|
|
")\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.naive_bayes import MultinomialNB\n",
|
|
"from sklearn.pipeline import Pipeline\n",
|
|
"from sklearn.svm import LinearSVC\n",
|
|
"\n",
|
|
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
|
|
"RANDOM_STATE = 42\n",
|
|
"\n",
|
|
"if not DATA_PATH.exists():\n",
|
|
" raise FileNotFoundError(\n",
|
|
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eae4780b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Prepare one fixed experiment\n",
|
|
"\n",
|
|
"Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6d147533",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
|
|
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
|
|
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
|
|
"df = df.dropna(subset=[\"label\", \"message\"])\n",
|
|
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
|
|
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
|
|
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
|
|
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
|
|
"\n",
|
|
"X = df[\"message\"]\n",
|
|
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
|
|
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
|
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
|
|
")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "899ce6a8",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Define the candidate pipelines\n",
|
|
"\n",
|
|
"All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7ac547e0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def make_pipeline(model):\n",
|
|
" return Pipeline([\n",
|
|
" (\"tfidf\", TfidfVectorizer(\n",
|
|
" lowercase=True,\n",
|
|
" ngram_range=(1, 2),\n",
|
|
" min_df=1,\n",
|
|
" sublinear_tf=True,\n",
|
|
" )),\n",
|
|
" (\"model\", model),\n",
|
|
" ])\n",
|
|
"\n",
|
|
"\n",
|
|
"models = {\n",
|
|
" \"MultinomialNB\": make_pipeline(MultinomialNB(alpha=1.0)),\n",
|
|
" \"Perceptron\": make_pipeline(Perceptron(\n",
|
|
" max_iter=2000,\n",
|
|
" tol=1e-3,\n",
|
|
" random_state=RANDOM_STATE,\n",
|
|
" )),\n",
|
|
" \"LinearSVC\": make_pipeline(LinearSVC(random_state=RANDOM_STATE)),\n",
|
|
" \"LogisticRegression\": make_pipeline(LogisticRegression(\n",
|
|
" max_iter=2000,\n",
|
|
" random_state=RANDOM_STATE,\n",
|
|
" )),\n",
|
|
"}\n",
|
|
"\n",
|
|
"print(\"Models:\", list(models))\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ebbc5289",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Train and evaluate fairly\n",
|
|
"\n",
|
|
"Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cc716625",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"fitted_models = {}\n",
|
|
"predictions = {}\n",
|
|
"rows = []\n",
|
|
"\n",
|
|
"for model_name, pipeline in models.items():\n",
|
|
" pipeline.fit(X_train, y_train)\n",
|
|
" y_pred = pipeline.predict(X_test)\n",
|
|
" cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
|
|
" tn, fp, fn, tp = cm.ravel()\n",
|
|
"\n",
|
|
" fitted_models[model_name] = pipeline\n",
|
|
" predictions[model_name] = y_pred\n",
|
|
" rows.append({\n",
|
|
" \"model\": model_name,\n",
|
|
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
|
|
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
|
|
" \"false_positives\": int(fp),\n",
|
|
" \"false_negatives\": int(fn),\n",
|
|
" })\n",
|
|
"\n",
|
|
"comparison = (\n",
|
|
" pd.DataFrame(rows)\n",
|
|
" .set_index(\"model\")\n",
|
|
" .sort_values([\"spam_f1\", \"spam_recall\"], ascending=False)\n",
|
|
")\n",
|
|
"comparison.round(4)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e6d4d8a6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Select and inspect the strongest model\n",
|
|
"\n",
|
|
"For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "86f3ebb4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"best_model_name = comparison[\"spam_f1\"].idxmax()\n",
|
|
"best_pipeline = fitted_models[best_model_name]\n",
|
|
"best_predictions = predictions[best_model_name]\n",
|
|
"\n",
|
|
"best_errors = pd.DataFrame({\n",
|
|
" \"message\": X_test.to_numpy(),\n",
|
|
" \"actual\": y_test.to_numpy(),\n",
|
|
" \"predicted\": best_predictions,\n",
|
|
"})\n",
|
|
"best_errors = best_errors[best_errors[\"actual\"].ne(best_errors[\"predicted\"])].copy()\n",
|
|
"best_errors[\"error_type\"] = np.where(\n",
|
|
" (best_errors[\"actual\"] == 0) & (best_errors[\"predicted\"] == 1),\n",
|
|
" \"false_positive\",\n",
|
|
" \"false_negative\",\n",
|
|
")\n",
|
|
"\n",
|
|
"print(\"Best model by spam F1:\", best_model_name)\n",
|
|
"print(\"Number of errors:\", len(best_errors))\n",
|
|
"best_errors.head(10)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a6dbf81d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Interpretation checkpoint\n",
|
|
"\n",
|
|
"A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a4c51d44",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Self-check\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2bd14558",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
|
|
"assert set(fitted_models) == set(models)\n",
|
|
"assert set(predictions) == set(models)\n",
|
|
"assert comparison.shape == (4, 6)\n",
|
|
"assert comparison.index.is_unique\n",
|
|
"assert comparison[\"spam_f1\"].between(0, 1).all()\n",
|
|
"assert best_model_name in models\n",
|
|
"assert len(best_predictions) == len(y_test)\n",
|
|
"assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
|
|
"print(\"✓ Lab 4 completed\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "19c2b755",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Suggested answers\n",
|
|
"\n",
|
|
"1. **Why must every model use the same data split?** \n",
|
|
" Changing both the split and model would confound the comparison. A fixed split isolates the effect of the classifier and makes the scores directly comparable.\n",
|
|
"2. **Why is a linear SVM appropriate for TF-IDF features?** \n",
|
|
" TF-IDF creates many sparse features. Linear SVM can learn a separating hyperplane efficiently in this high-dimensional space without calculating an expensive nonlinear kernel matrix.\n",
|
|
"3. **Why might the highest-accuracy model still be unsuitable for spam filtering?** \n",
|
|
" Accuracy can be dominated by the majority ham class. A model may still miss too much spam or block too many legitimate messages, which is visible in spam recall, precision, and error counts.\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|