mikrotik/labs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Browse Source
This commit is contained in:
mikrotik
2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Lab-4/Lab-4-Solution/01_SMS_Dataset_Inspection_Solution.ipynb
+252
View File
@@ -0,0 +1,252 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "979aed09",
"metadata": {},
"source": [
"# Module 4 — Lab 1: Inspecting the SMS Spam Dataset\n",
"\n",
"\n",
"**Topics:** text data, duplicates, conflicting labels, class distribution\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "c9b13285",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load the original UCI tab-separated text file;\n",
"- audit missing values, exact duplicates, and conflicting labels;\n",
"- create a binary target without modifying the message text;\n",
"- describe class imbalance and message-length differences;\n",
"- explain why aggressive cleaning may remove useful security signals.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8c2b2a4a",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download \"\n",
" \"and extract the official UCI SMS Spam Collection.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "184f1a06",
"metadata": {},
"source": [
"## 1. Load the original collection\n",
"\n",
"The source file has no header. Each row contains a label, one tab character, and the message text.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f376803d",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(\n",
" DATA_PATH,\n",
" sep=\"\\t\",\n",
" header=None,\n",
" names=[\"label\", \"message\"],\n",
" encoding=\"utf-8\",\n",
")\n",
"\n",
"print(\"Raw shape:\", raw_df.shape)\n",
"print(raw_df[\"label\"].value_counts(dropna=False))\n",
"raw_df.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "4ae84500",
"metadata": {},
"source": [
"## 2. Audit the records\n",
"\n",
"Exact duplicates can cross a random train-test split and make evaluation too optimistic. A more serious problem occurs when identical text has different labels. Audit both conditions before modeling.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "177c72a1",
"metadata": {},
"outputs": [],
"source": [
"audited_df = raw_df.copy()\n",
"audited_df[\"label\"] = audited_df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"audited_df[\"message\"] = audited_df[\"message\"].astype(\"string\").str.strip()\n",
"audited_df = audited_df.dropna(subset=[\"label\", \"message\"])\n",
"audited_df = audited_df[audited_df[\"message\"].ne(\"\")]\n",
"audited_df = audited_df[audited_df[\"label\"].isin([\"ham\", \"spam\"])]\n",
"\n",
"exact_duplicate_count = int(audited_df.duplicated(subset=[\"label\", \"message\"]).sum())\n",
"label_counts_per_message = audited_df.groupby(\"message\")[\"label\"].nunique()\n",
"conflicting_messages = label_counts_per_message[label_counts_per_message.gt(1)].index\n",
"\n",
"print(\"Exact duplicate rows:\", exact_duplicate_count)\n",
"print(\"Conflicting message texts:\", len(conflicting_messages))\n"
]
},
{
"cell_type": "markdown",
"id": "b9d41a33",
"metadata": {},
"source": [
"## 3. Build the clean classroom table\n",
"\n",
"Remove ambiguous messages before exact duplicates, then map `ham → 0` and `spam → 1`. Keep the original text intact.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0b915b1",
"metadata": {},
"outputs": [],
"source": [
"clean_df = audited_df[~audited_df[\"message\"].isin(conflicting_messages)].copy()\n",
"clean_df = clean_df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"clean_df[\"target\"] = clean_df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"print(\"Clean shape:\", clean_df.shape)\n",
"print(clean_df[\"target\"].value_counts().sort_index().rename(index={0: \"ham\", 1: \"spam\"}))\n"
]
},
{
"cell_type": "markdown",
"id": "74762f79",
"metadata": {},
"source": [
"## 4. Describe the classes and message lengths\n",
"\n",
"Length is used only for exploration. The later models will learn from the message text itself.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1334e9da",
"metadata": {},
"outputs": [],
"source": [
"clean_df[\"character_count\"] = clean_df[\"message\"].str.len()\n",
"clean_df[\"word_count\"] = clean_df[\"message\"].str.split().str.len()\n",
"\n",
"length_summary = clean_df.groupby(\"label\").agg(\n",
" message_count=(\"message\", \"size\"),\n",
" median_characters=(\"character_count\", \"median\"),\n",
" median_words=(\"word_count\", \"median\"),\n",
")\n",
"class_distribution = clean_df[\"label\"].value_counts(normalize=True)\n",
"\n",
"print(length_summary)\n",
"print(\"Class proportions:\")\n",
"print(class_distribution)\n"
]
},
{
"cell_type": "markdown",
"id": "34c637b9",
"metadata": {},
"source": [
"## Security note: do not over-clean\n",
"\n",
"URLs, phone numbers, currency symbols, punctuation, unusual capitalization, and deliberate misspellings may contain spam evidence. Begin with the vectorizer defaults. Add normalization only when an experiment demonstrates that it improves validation performance without hiding operationally important signals.\n"
]
},
{
"cell_type": "markdown",
"id": "27f4bb63",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "870a2fbe",
"metadata": {},
"outputs": [],
"source": [
"assert list(raw_df.columns) == [\"label\", \"message\"]\n",
"assert clean_df[\"message\"].notna().all()\n",
"assert clean_df[\"message\"].ne(\"\").all()\n",
"assert set(clean_df[\"label\"].unique()) == {\"ham\", \"spam\"}\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert not clean_df[\"message\"].isin(conflicting_messages).any()\n",
"assert class_distribution.loc[\"ham\"] > class_distribution.loc[\"spam\"]\n",
"assert {\"character_count\", \"word_count\"}.issubset(clean_df.columns)\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "3a766ad5",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why should exact duplicates be removed before the train-test split?** \n",
" A duplicate can otherwise appear in both partitions, allowing the model to receive nearly identical training and test evidence and inflating the reported score.\n",
"2. **Why might removing URLs and currency symbols reduce detection quality?** \n",
" Those tokens often carry direct spam intent. Removing them can discard discriminative evidence rather than remove harmless noise.\n",
"3. **Why is accuracy alone risky for this dataset?** \n",
" The ham class is much larger, so a model can obtain high accuracy while missing many spam messages. Spam precision, recall, F1, and the confusion matrix are also required.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcda6cf6-3ba4-4229-8991-8efb2209d294",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/02_Bag_of_Words_TFIDF_Solution.ipynb
+273
View File
@@ -0,0 +1,273 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "942ac4eb",
"metadata": {},
"source": [
"# Module 4 — Lab 2: Bag of Words and TF-IDF\n",
"\n",
"\n",
"**Topics:** CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "554dc6ef",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- explain how Bag of Words maps documents to numeric vectors;\n",
"- inspect a document-term matrix and its vocabulary;\n",
"- fit TF-IDF on training text and transform unseen text;\n",
"- describe why text matrices are high-dimensional and sparse;\n",
"- compare unigram features with unigram-plus-bigram features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dee543fe",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "637b0eaa",
"metadata": {},
"source": [
"## 1. Make Bag of Words visible\n",
"\n",
"Use three short messages so that every row, token, and count can be inspected directly.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c3d5152",
"metadata": {},
"outputs": [],
"source": [
"tiny_messages = [\n",
" \"free prize now\",\n",
" \"free meeting today\",\n",
" \"project meeting today\",\n",
"]\n",
"\n",
"count_vectorizer = CountVectorizer()\n",
"tiny_bow = count_vectorizer.fit_transform(tiny_messages)\n",
"tiny_bow_df = pd.DataFrame(\n",
" tiny_bow.toarray(),\n",
" columns=count_vectorizer.get_feature_names_out(),\n",
" index=[\"message_1\", \"message_2\", \"message_3\"],\n",
")\n",
"tiny_bow_df\n"
]
},
{
"cell_type": "markdown",
"id": "e3e98734",
"metadata": {},
"source": [
"## 2. Prepare the UCI messages\n",
"\n",
"Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9841f9b7",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"\n",
"conflicts = df.groupby(\"message\")[\"label\"].nunique()\n",
"conflict_texts = conflicts[conflicts.gt(1)].index\n",
"df = df[~df[\"message\"].isin(conflict_texts)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"print(\"Clean messages:\", len(df))\n"
]
},
{
"cell_type": "markdown",
"id": "92848cea",
"metadata": {},
"source": [
"## 3. Fit TF-IDF without leakage\n",
"\n",
"`fit_transform` learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only `transform`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e5a89189",
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.20,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"\n",
"tfidf_vectorizer = TfidfVectorizer(lowercase=True)\n",
"X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)\n",
"X_test_tfidf = tfidf_vectorizer.transform(X_test)\n",
"\n",
"density = X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1])\n",
"print(\"Training matrix shape:\", X_train_tfidf.shape)\n",
"print(\"Test matrix shape:\", X_test_tfidf.shape)\n",
"print(\"Training density:\", f\"{density:.4%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "a4230176",
"metadata": {},
"source": [
"## 4. Inspect the learned feature space\n",
"\n",
"A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "90e679d0",
"metadata": {},
"outputs": [],
"source": [
"mean_tfidf = np.asarray(X_train_tfidf.mean(axis=0)).ravel()\n",
"top_terms = pd.Series(\n",
" mean_tfidf,\n",
" index=tfidf_vectorizer.get_feature_names_out(),\n",
" name=\"mean_tfidf\",\n",
").sort_values(ascending=False).head(15)\n",
"top_terms\n"
]
},
{
"cell_type": "markdown",
"id": "2ebe7eee",
"metadata": {},
"source": [
"## 5. Add short phrases with bigrams\n",
"\n",
"A unigram represents one token. A bigram can preserve short phrases such as `call now` or `free entry`, but it also increases the number of features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c1de19b",
"metadata": {},
"outputs": [],
"source": [
"ngram_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1)\n",
"X_train_ngram = ngram_vectorizer.fit_transform(X_train)\n",
"ngram_features = ngram_vectorizer.get_feature_names_out()\n",
"example_bigrams = [term for term in ngram_features if \" \" in term][:10]\n",
"\n",
"print(\"Unigram features:\", X_train_tfidf.shape[1])\n",
"print(\"Unigram + bigram features:\", X_train_ngram.shape[1])\n",
"print(\"Example bigrams:\", example_bigrams)\n"
]
},
{
"cell_type": "markdown",
"id": "64432265",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36f18994",
"metadata": {},
"outputs": [],
"source": [
"assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))\n",
"assert tiny_bow_df.loc[\"message_1\", \"free\"] == 1\n",
"assert len(X_train) + len(X_test) == len(df)\n",
"assert X_train_tfidf.shape[0] == len(X_train)\n",
"assert X_test_tfidf.shape[0] == len(X_test)\n",
"assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]\n",
"assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]\n",
"assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]\n",
"assert len(top_terms) == 15\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "991fa410",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is a text feature matrix usually sparse?** \n",
" Each message uses only a small fraction of the complete vocabulary, so most document-term cells are zero.\n",
"2. **Why must the vectorizer be fitted only on training messages?** \n",
" Fitting on all messages reveals test-set vocabulary and frequency information during training, which is data leakage and can make evaluation optimistic.\n",
"3. **What trade-off is introduced by adding bigrams?** \n",
" Bigrams preserve useful short phrases but expand the feature space, consume more memory, and can overfit when phrases are rare.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/03_Naive_Bayes_Baseline_Solution.ipynb
+294
View File
@@ -0,0 +1,294 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "14ac1890",
"metadata": {},
"source": [
"# Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline\n",
"\n",
"**Topics:** Pipeline, MultinomialNB, confusion matrix, baseline, error analysis\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "af2dc7be",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- place TF-IDF and Multinomial Naive Bayes in one pipeline;\n",
"- explain how the pipeline prevents preprocessing leakage;\n",
"- calculate spam precision, recall, and F1;\n",
"- compare a trained classifier with a naive baseline;\n",
"- inspect false positives and false negatives.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bced8aa9",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "aa622049",
"metadata": {},
"source": [
"## 1. Load, audit, and split the messages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9efd4a80",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"print(\"Training messages:\", len(X_train))\n",
"print(\"Test messages:\", len(X_test))\n",
"print(\"Training spam rate:\", f\"{y_train.mean():.2%}\")\n",
"print(\"Test spam rate:\", f\"{y_test.mean():.2%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "263fa9cf",
"metadata": {},
"source": [
"## 2. Build and train one pipeline\n",
"\n",
"The pipeline fits TF-IDF only on `X_train`. During prediction, it applies the already-fitted transformation to `X_test`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aed27122",
"metadata": {},
"outputs": [],
"source": [
"nb_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(lowercase=True, ngram_range=(1, 1))),\n",
" (\"model\", MultinomialNB(alpha=1.0)),\n",
"])\n",
"nb_pipeline.fit(X_train, y_train)\n",
"y_pred = nb_pipeline.predict(X_test)\n",
"\n",
"print(nb_pipeline)\n"
]
},
{
"cell_type": "markdown",
"id": "50d5bc04",
"metadata": {},
"source": [
"## 3. Evaluate the spam class\n",
"\n",
"- **Spam precision:** when the model blocks a message, how often is it truly spam?\n",
"- **Spam recall:** of all spam messages, how many did the model catch?\n",
"- **Spam F1:** harmonic mean of spam precision and recall.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32b32feb",
"metadata": {},
"outputs": [],
"source": [
"cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
"tn, fp, fn, tp = cm.ravel()\n",
"\n",
"nb_metrics = {\n",
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
"}\n",
"\n",
"print(pd.Series(nb_metrics).round(4))\n",
"print(\"\\nClassification report:\")\n",
"print(classification_report(y_test, y_pred, target_names=[\"ham\", \"spam\"], zero_division=0))\n"
]
},
{
"cell_type": "markdown",
"id": "3a7568ba",
"metadata": {},
"source": [
"## 4. Compare with a naive baseline\n",
"\n",
"Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7198e0cf",
"metadata": {},
"outputs": [],
"source": [
"dummy = DummyClassifier(strategy=\"most_frequent\")\n",
"dummy.fit(X_train.to_frame(), y_train)\n",
"dummy_pred = dummy.predict(X_test.to_frame())\n",
"\n",
"comparison = pd.DataFrame([\n",
" {\n",
" \"model\": \"Most-frequent baseline\",\n",
" \"accuracy\": accuracy_score(y_test, dummy_pred),\n",
" \"spam_f1\": f1_score(y_test, dummy_pred, pos_label=1, zero_division=0),\n",
" },\n",
" {\n",
" \"model\": \"TF-IDF + MultinomialNB\",\n",
" \"accuracy\": nb_metrics[\"accuracy\"],\n",
" \"spam_f1\": nb_metrics[\"spam_f1\"],\n",
" },\n",
"]).set_index(\"model\")\n",
"comparison\n"
]
},
{
"cell_type": "markdown",
"id": "56233ccf",
"metadata": {},
"source": [
"## 5. Inspect operational errors\n",
"\n",
"A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e601a648",
"metadata": {},
"outputs": [],
"source": [
"error_analysis = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual\": y_test.to_numpy(),\n",
" \"predicted\": y_pred,\n",
"})\n",
"errors = error_analysis[error_analysis[\"actual\"].ne(error_analysis[\"predicted\"])].copy()\n",
"errors[\"error_type\"] = np.where(\n",
" (errors[\"actual\"] == 0) & (errors[\"predicted\"] == 1),\n",
" \"false_positive\",\n",
" \"false_negative\",\n",
")\n",
"\n",
"print(errors[\"error_type\"].value_counts())\n",
"errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "a818ba0b",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "72c65672",
"metadata": {},
"outputs": [],
"source": [
"assert list(nb_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert isinstance(nb_pipeline.named_steps[\"model\"], MultinomialNB)\n",
"assert len(y_pred) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert set(nb_metrics) == {\"accuracy\", \"spam_precision\", \"spam_recall\", \"spam_f1\"}\n",
"assert comparison.loc[\"Most-frequent baseline\", \"spam_f1\"] == 0\n",
"assert set(errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "7f0d0753",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is the most-frequent classifier an important baseline?** \n",
" It reveals how much accuracy comes from class imbalance alone. A useful spam detector must outperform the baseline on spam-sensitive metrics.\n",
"2. **Which operational harm is represented by a false positive?** \n",
" A legitimate message is incorrectly blocked or quarantined, which can interrupt communication and reduce user trust.\n",
"3. **Why should TF-IDF and the model be placed in one pipeline?** \n",
" The pipeline guarantees that vocabulary and IDF statistics are learned only during fitting and then reused consistently, reducing leakage and preprocessing mistakes.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/04_Linear_Model_Comparison_Solution.ipynb
+304
View File
@@ -0,0 +1,304 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d43eebb6",
"metadata": {},
"source": [
"# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
"\n",
"\n",
"**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "78892370",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare several classifiers on one fixed train-test split;\n",
"- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
"- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
"- connect false positives and false negatives to operational consequences;\n",
"- explain why kernel SVM is not required for this classroom workflow.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d9e93987",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression, Perceptron\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.svm import LinearSVC\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "eae4780b",
"metadata": {},
"source": [
"## 1. Prepare one fixed experiment\n",
"\n",
"Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6d147533",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "899ce6a8",
"metadata": {},
"source": [
"## 2. Define the candidate pipelines\n",
"\n",
"All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7ac547e0",
"metadata": {},
"outputs": [],
"source": [
"def make_pipeline(model):\n",
" return Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", model),\n",
" ])\n",
"\n",
"\n",
"models = {\n",
" \"MultinomialNB\": make_pipeline(MultinomialNB(alpha=1.0)),\n",
" \"Perceptron\": make_pipeline(Perceptron(\n",
" max_iter=2000,\n",
" tol=1e-3,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
" \"LinearSVC\": make_pipeline(LinearSVC(random_state=RANDOM_STATE)),\n",
" \"LogisticRegression\": make_pipeline(LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"}\n",
"\n",
"print(\"Models:\", list(models))\n"
]
},
{
"cell_type": "markdown",
"id": "ebbc5289",
"metadata": {},
"source": [
"## 3. Train and evaluate fairly\n",
"\n",
"Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc716625",
"metadata": {},
"outputs": [],
"source": [
"fitted_models = {}\n",
"predictions = {}\n",
"rows = []\n",
"\n",
"for model_name, pipeline in models.items():\n",
" pipeline.fit(X_train, y_train)\n",
" y_pred = pipeline.predict(X_test)\n",
" cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
" tn, fp, fn, tp = cm.ravel()\n",
"\n",
" fitted_models[model_name] = pipeline\n",
" predictions[model_name] = y_pred\n",
" rows.append({\n",
" \"model\": model_name,\n",
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" })\n",
"\n",
"comparison = (\n",
" pd.DataFrame(rows)\n",
" .set_index(\"model\")\n",
" .sort_values([\"spam_f1\", \"spam_recall\"], ascending=False)\n",
")\n",
"comparison.round(4)\n"
]
},
{
"cell_type": "markdown",
"id": "e6d4d8a6",
"metadata": {},
"source": [
"## 4. Select and inspect the strongest model\n",
"\n",
"For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "86f3ebb4",
"metadata": {},
"outputs": [],
"source": [
"best_model_name = comparison[\"spam_f1\"].idxmax()\n",
"best_pipeline = fitted_models[best_model_name]\n",
"best_predictions = predictions[best_model_name]\n",
"\n",
"best_errors = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual\": y_test.to_numpy(),\n",
" \"predicted\": best_predictions,\n",
"})\n",
"best_errors = best_errors[best_errors[\"actual\"].ne(best_errors[\"predicted\"])].copy()\n",
"best_errors[\"error_type\"] = np.where(\n",
" (best_errors[\"actual\"] == 0) & (best_errors[\"predicted\"] == 1),\n",
" \"false_positive\",\n",
" \"false_negative\",\n",
")\n",
"\n",
"print(\"Best model by spam F1:\", best_model_name)\n",
"print(\"Number of errors:\", len(best_errors))\n",
"best_errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "a6dbf81d",
"metadata": {},
"source": [
"## Interpretation checkpoint\n",
"\n",
"A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
]
},
{
"cell_type": "markdown",
"id": "a4c51d44",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bd14558",
"metadata": {},
"outputs": [],
"source": [
"assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
"assert set(fitted_models) == set(models)\n",
"assert set(predictions) == set(models)\n",
"assert comparison.shape == (4, 6)\n",
"assert comparison.index.is_unique\n",
"assert comparison[\"spam_f1\"].between(0, 1).all()\n",
"assert best_model_name in models\n",
"assert len(best_predictions) == len(y_test)\n",
"assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "19c2b755",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why must every model use the same data split?** \n",
" Changing both the split and model would confound the comparison. A fixed split isolates the effect of the classifier and makes the scores directly comparable.\n",
"2. **Why is a linear SVM appropriate for TF-IDF features?** \n",
" TF-IDF creates many sparse features. Linear SVM can learn a separating hyperplane efficiently in this high-dimensional space without calculating an expensive nonlinear kernel matrix.\n",
"3. **Why might the highest-accuracy model still be unsuitable for spam filtering?** \n",
" Accuracy can be dominated by the majority ham class. A model may still miss too much spam or block too many legitimate messages, which is visible in spam recall, precision, and error counts.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/05_Probability_Thresholds_Interpretation_Solution.ipynb
+362
View File
@@ -0,0 +1,362 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b438e257",
"metadata": {},
"source": [
"# Module 4 — Lab 5: Probability, Thresholds, and Interpretation\n",
"\n",
"\n",
"**Topics:** Logistic Regression, predict_proba, validation, thresholds, coefficients, actions\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "e52f8f6f",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- reserve separate training, validation, and test partitions;\n",
"- convert Logistic Regression probabilities into decisions;\n",
"- choose a threshold using validation data only;\n",
"- measure the threshold trade-off between false positives and false negatives;\n",
"- interpret influential terms through model coefficients;\n",
"- map model scores to allow, analyst-review, and quarantine actions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "840495c7",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"THRESHOLDS = np.arange(0.20, 0.81, 0.05)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "a4aa5a16",
"metadata": {},
"source": [
"## 1. Prepare train, validation, and test data\n",
"\n",
"Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47bfe88f",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cf3ba33b",
"metadata": {},
"outputs": [],
"source": [
"X_train, X_temp, y_train, y_temp = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.40,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"X_valid, X_test, y_valid, y_test = train_test_split(\n",
" X_temp,\n",
" y_temp,\n",
" test_size=0.50,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y_temp,\n",
")\n",
"\n",
"print(\"Train / validation / test:\", len(X_train), len(X_valid), len(X_test))\n",
"print(\"Spam rates:\", f\"{y_train.mean():.2%}\", f\"{y_valid.mean():.2%}\", f\"{y_test.mean():.2%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "82a1cadd",
"metadata": {},
"source": [
"## 2. Train a probabilistic text classifier\n",
"\n",
"Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ecb39196",
"metadata": {},
"outputs": [],
"source": [
"lr_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"])\n",
"lr_pipeline.fit(X_train, y_train)\n",
"valid_probabilities = lr_pipeline.predict_proba(X_valid)[:, 1]\n",
"\n",
"print(\"First probabilities:\", np.round(valid_probabilities[:10], 3))\n"
]
},
{
"cell_type": "markdown",
"id": "b2e41b60",
"metadata": {},
"source": [
"## 3. Select a threshold on validation data\n",
"\n",
"The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "54cf81c8",
"metadata": {},
"outputs": [],
"source": [
"rows = []\n",
"for threshold in THRESHOLDS:\n",
" predictions = (valid_probabilities >= threshold).astype(int)\n",
" cm = confusion_matrix(y_valid, predictions, labels=[0, 1])\n",
" tn, fp, fn, tp = cm.ravel()\n",
" rows.append({\n",
" \"threshold\": threshold,\n",
" \"spam_precision\": precision_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" })\n",
"\n",
"threshold_table = pd.DataFrame(rows).set_index(\"threshold\")\n",
"eligible = threshold_table[threshold_table[\"spam_recall\"].ge(0.80)]\n",
"if eligible.empty:\n",
" selected_threshold = float(threshold_table[\"spam_f1\"].idxmax())\n",
"else:\n",
" selected_threshold = float(\n",
" eligible.sort_values([\"spam_precision\", \"spam_f1\"], ascending=False).index[0]\n",
" )\n",
"\n",
"print(threshold_table.round(4))\n",
"print(\"Selected threshold:\", selected_threshold)\n"
]
},
{
"cell_type": "markdown",
"id": "b1658a9c",
"metadata": {},
"source": [
"## 4. Evaluate once on the untouched test set\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f5d43c65",
"metadata": {},
"outputs": [],
"source": [
"test_probabilities = lr_pipeline.predict_proba(X_test)[:, 1]\n",
"test_predictions = (test_probabilities >= selected_threshold).astype(int)\n",
"test_cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n",
"tn, fp, fn, tp = test_cm.ravel()\n",
"test_metrics = {\n",
" \"threshold\": selected_threshold,\n",
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
" \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
"}\n",
"\n",
"print(pd.Series(test_metrics).round(4))\n",
"print(test_cm)\n"
]
},
{
"cell_type": "markdown",
"id": "f41cbbb5",
"metadata": {},
"source": [
"## 5. Interpret influential terms\n",
"\n",
"Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "87ea0bc0",
"metadata": {},
"outputs": [],
"source": [
"feature_names = lr_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n",
"coefficients = lr_pipeline.named_steps[\"model\"].coef_[0]\n",
"coefficient_series = pd.Series(coefficients, index=feature_names)\n",
"\n",
"top_ham = coefficient_series.nsmallest(15).rename(\"coefficient\").reset_index()\n",
"top_ham.columns = [\"term\", \"coefficient\"]\n",
"top_ham[\"direction\"] = \"ham\"\n",
"\n",
"top_spam = coefficient_series.nlargest(15).rename(\"coefficient\").reset_index()\n",
"top_spam.columns = [\"term\", \"coefficient\"]\n",
"top_spam[\"direction\"] = \"spam\"\n",
"\n",
"top_features = pd.concat([top_spam, top_ham], ignore_index=True)\n",
"top_features\n"
]
},
{
"cell_type": "markdown",
"id": "7705ca0e",
"metadata": {},
"source": [
"## 6. Map probabilities to actions\n",
"\n",
"Use fixed demonstration bands: below 0.30 is `allow`, 0.30–0.70 is `analyst_review`, and 0.70 or above is `quarantine`. A production policy would set these boundaries from business cost, capacity, and validation evidence.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a32d7b4",
"metadata": {},
"outputs": [],
"source": [
"policy_actions = pd.cut(\n",
" test_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
")\n",
"action_summary = pd.crosstab(\n",
" pd.Series(y_test.to_numpy(), name=\"actual\"),\n",
" pd.Series(policy_actions, name=\"action\"),\n",
").rename(index={0: \"ham\", 1: \"spam\"})\n",
"action_summary\n"
]
},
{
"cell_type": "markdown",
"id": "d9e5fb09",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3da1834c",
"metadata": {},
"outputs": [],
"source": [
"assert len(X_train) + len(X_valid) + len(X_test) == len(df)\n",
"assert set(lr_pipeline.named_steps) == {\"tfidf\", \"model\"}\n",
"assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))\n",
"assert threshold_table.index.is_monotonic_increasing\n",
"assert selected_threshold in threshold_table.index\n",
"assert len(test_predictions) == len(y_test)\n",
"assert test_cm.shape == (2, 2)\n",
"assert len(top_features) == 30\n",
"assert set(top_features[\"direction\"]) == {\"ham\", \"spam\"}\n",
"assert set(policy_actions.astype(\"string\").unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"print(\"✓ Lab 5 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "d681ad09",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is a validation set needed when selecting a threshold?** \n",
" Using test labels to choose the threshold would tune the policy to the final evaluation data. Validation data supports selection while keeping the test estimate independent.\n",
"2. **What usually happens to spam recall when the threshold is lowered?** \n",
" More messages exceed the lower threshold, so spam recall usually rises, but false positives may also increase and precision may fall.\n",
"3. **Why are large positive coefficients not proof that a term causes spam?** \n",
" A coefficient records an association learned from this corpus and representation. It may reflect dataset-specific patterns, confounding, or shortcuts rather than causation.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/06_Capstone_SMS_Spam_Detector_Solution.ipynb
+456
View File
@@ -0,0 +1,456 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4116df5d",
"metadata": {},
"source": [
"# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n",
"\n",
" \n",
"**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "81a0abfa",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build one complete, reproducible, and leakage-safe spam-detection workflow:\n",
"\n",
"1. load and audit the UCI SMS Spam Collection;\n",
"2. create one stratified holdout test set;\n",
"3. train a word-level TF-IDF and Logistic Regression pipeline;\n",
"4. evaluate spam-sensitive metrics and operational errors;\n",
"5. inspect influential terms and test custom messages;\n",
"6. export evidence and an analyst-facing summary.\n",
"\n",
"The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b945ce17",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import textwrap\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay,\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"RANDOM_STATE = 42\n",
"DECISION_THRESHOLD = 0.50\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n",
"\n",
"OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n"
]
},
{
"cell_type": "markdown",
"id": "8ea5e886",
"metadata": {},
"source": [
"## 1. Load, audit, and split the data\n",
"\n",
"Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b604f8da",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"clean_df = raw_df.copy()\n",
"clean_df[\"label\"] = clean_df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"clean_df[\"message\"] = clean_df[\"message\"].astype(\"string\").str.strip()\n",
"clean_df = clean_df.dropna(subset=[\"label\", \"message\"])\n",
"clean_df = clean_df[\n",
" clean_df[\"label\"].isin([\"ham\", \"spam\"]) & clean_df[\"message\"].ne(\"\")\n",
"]\n",
"conflict_counts = clean_df.groupby(\"message\")[\"label\"].nunique()\n",
"conflict_texts = conflict_counts[conflict_counts.gt(1)].index\n",
"clean_df = clean_df[~clean_df[\"message\"].isin(conflict_texts)]\n",
"clean_df = clean_df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"clean_df[\"target\"] = clean_df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"X = clean_df[\"message\"]\n",
"y = clean_df[\"target\"]\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.20,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"\n",
"print(\"Raw rows:\", len(raw_df))\n",
"print(\"Clean rows:\", len(clean_df))\n",
"print(\"Training / test:\", len(X_train), len(X_test))\n"
]
},
{
"cell_type": "markdown",
"id": "578afb0e",
"metadata": {},
"source": [
"## 2. Build and train the final pipeline\n",
"\n",
"The pipeline learns its vocabulary and IDF weights from the training messages only.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b9eafd46",
"metadata": {},
"outputs": [],
"source": [
"final_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"])\n",
"final_pipeline.fit(X_train, y_train)\n",
"test_probabilities = final_pipeline.predict_proba(X_test)[:, 1]\n",
"test_predictions = (test_probabilities >= DECISION_THRESHOLD).astype(int)\n",
"\n",
"print(\"Predictions created:\", len(test_predictions))\n"
]
},
{
"cell_type": "markdown",
"id": "7b6ed943",
"metadata": {},
"source": [
"## 3. Evaluate the held-out test set\n",
"\n",
"Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7faddf17",
"metadata": {},
"outputs": [],
"source": [
"cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n",
"tn, fp, fn, tp = cm.ravel()\n",
"metrics = {\n",
" \"decision_threshold\": DECISION_THRESHOLD,\n",
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
" \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"true_negatives\": int(tn),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"true_positives\": int(tp),\n",
"}\n",
"\n",
"print(pd.Series(metrics).round(4))\n",
"print(classification_report(\n",
" y_test,\n",
" test_predictions,\n",
" target_names=[\"ham\", \"spam\"],\n",
" zero_division=0,\n",
"))\n"
]
},
{
"cell_type": "markdown",
"id": "57c486cb",
"metadata": {},
"source": [
"## 4. Create an analyst error table\n",
"\n",
"Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d612fce7",
"metadata": {},
"outputs": [],
"source": [
"actions = pd.cut(\n",
" test_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
").astype(\"string\")\n",
"\n",
"prediction_table = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual_label\": np.where(y_test.to_numpy() == 1, \"spam\", \"ham\"),\n",
" \"predicted_label\": np.where(test_predictions == 1, \"spam\", \"ham\"),\n",
" \"spam_probability\": test_probabilities,\n",
" \"action\": actions,\n",
"})\n",
"prediction_table[\"error_type\"] = np.select(\n",
" [\n",
" (prediction_table[\"actual_label\"] == \"ham\") & (prediction_table[\"predicted_label\"] == \"spam\"),\n",
" (prediction_table[\"actual_label\"] == \"spam\") & (prediction_table[\"predicted_label\"] == \"ham\"),\n",
" ],\n",
" [\"false_positive\", \"false_negative\"],\n",
" default=\"correct\",\n",
")\n",
"error_table = prediction_table[prediction_table[\"error_type\"].ne(\"correct\")].copy()\n",
"\n",
"print(\"Errors by type:\")\n",
"print(error_table[\"error_type\"].value_counts())\n",
"error_table.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "99d1bba1",
"metadata": {},
"source": [
"## 5. Inspect influential words and phrases\n",
"\n",
"Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ebf3fde7",
"metadata": {},
"outputs": [],
"source": [
"feature_names = final_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n",
"coefficients = pd.Series(\n",
" final_pipeline.named_steps[\"model\"].coef_[0],\n",
" index=feature_names,\n",
" name=\"coefficient\",\n",
")\n",
"\n",
"top_spam = coefficients.nlargest(20).reset_index()\n",
"top_spam.columns = [\"term\", \"coefficient\"]\n",
"top_spam[\"direction\"] = \"spam\"\n",
"top_ham = coefficients.nsmallest(20).reset_index()\n",
"top_ham.columns = [\"term\", \"coefficient\"]\n",
"top_ham[\"direction\"] = \"ham\"\n",
"top_terms = pd.concat([top_spam, top_ham], ignore_index=True)\n",
"top_terms.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "779dc4d3",
"metadata": {},
"source": [
"## 6. Challenge the detector with new messages\n",
"\n",
"The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c27e573",
"metadata": {},
"outputs": [],
"source": [
"custom_messages = pd.Series([\n",
" \"Can we move the project meeting to 3 pm?\",\n",
" \"Congratulations! Claim your free cash prize now\",\n",
" \"URGENT: verify your account at the link below\",\n",
" \"You are a w1nner, reply FR33 to collect your reward\",\n",
" \"Please call me when you arrive at the station\",\n",
"], name=\"message\")\n",
"\n",
"custom_probabilities = final_pipeline.predict_proba(custom_messages)[:, 1]\n",
"custom_results = pd.DataFrame({\n",
" \"message\": custom_messages,\n",
" \"spam_probability\": custom_probabilities,\n",
" \"predicted_label\": np.where(custom_probabilities >= DECISION_THRESHOLD, \"spam\", \"ham\"),\n",
" \"action\": pd.cut(\n",
" custom_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
" ).astype(\"string\"),\n",
"})\n",
"custom_results\n"
]
},
{
"cell_type": "markdown",
"id": "a27debbf",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2d3eb2ed",
"metadata": {},
"outputs": [],
"source": [
"pd.Series(metrics, name=\"value\").to_csv(OUTPUT_DIR / \"capstone_metrics.csv\", header=True)\n",
"prediction_table.to_csv(OUTPUT_DIR / \"capstone_predictions.csv\", index=False)\n",
"top_terms.to_csv(OUTPUT_DIR / \"capstone_top_terms.csv\", index=False)\n",
"\n",
"fig, ax = plt.subplots(figsize=(5, 4))\n",
"ConfusionMatrixDisplay(\n",
" confusion_matrix=cm,\n",
" display_labels=[\"ham\", \"spam\"],\n",
").plot(ax=ax, cmap=\"Blues\", colorbar=False)\n",
"ax.set_title(\"SMS Spam Detector — Test Set\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"capstone_confusion_matrix.png\", dpi=160, bbox_inches=\"tight\")\n",
"plt.close(fig)\n",
"\n",
"summary = f'''SMS Spam Detector — Analyst Summary\n",
"===================================\n",
"Source dataset: UCI SMS Spam Collection\n",
"Clean rows: {len(clean_df)}\n",
"Training rows: {len(X_train)}\n",
"Test rows: {len(X_test)}\n",
"Overall spam rate: {y.mean():.2%}\n",
"Decision threshold: {DECISION_THRESHOLD:.2f}\n",
"\n",
"Test performance\n",
"----------------\n",
"Accuracy: {metrics['accuracy']:.3f}\n",
"Spam precision: {metrics['spam_precision']:.3f}\n",
"Spam recall: {metrics['spam_recall']:.3f}\n",
"Spam F1: {metrics['spam_f1']:.3f}\n",
"False positives: {fp}\n",
"False negatives: {fn}\n",
"\n",
"Operational interpretation\n",
"--------------------------\n",
"Messages below 0.30 are allowed, scores from 0.30 to below 0.70 are sent\n",
"for analyst review, and scores of 0.70 or more are quarantined.\n",
"\n",
"Limitation\n",
"----------\n",
"This classroom model uses SMS body text only. It does not inspect email headers,\n",
"sender domains, attachments, destination URLs, or changing attacker behavior.\n",
"'''\n",
"summary = textwrap.dedent(summary).strip() + \"\\n\"\n",
"(OUTPUT_DIR / \"capstone_summary.txt\").write_text(summary, encoding=\"utf-8\")\n",
"\n",
"print(summary)\n"
]
},
{
"cell_type": "markdown",
"id": "4ad1b775",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a650060",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_predictions.csv\",\n",
" \"capstone_top_terms.csv\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_summary.txt\",\n",
"}\n",
"\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert len(test_predictions) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert len(prediction_table) == len(y_test)\n",
"assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"assert len(top_terms) == 40\n",
"assert len(custom_results) == 5\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"Limitation\" in summary\n",
"print(\"✓ Lab 6 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "f680874c",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **What do the false positives and false negatives mean to a security team?** \n",
" False positives consume review effort or block legitimate communication. False negatives allow spam through and may expose users to fraud or malicious links. The acceptable balance depends on organizational cost and risk.\n",
"2. **Why should the custom obfuscated message be included in the capstone?** \n",
" It checks whether the learned word vocabulary is brittle when attackers alter spelling. A failure is useful evidence for considering character n-grams or additional signals later.\n",
"3. **What evidence is still missing before this classroom model could support real email security decisions?** \n",
" The model needs representative current email data, sender and header features, URL and attachment analysis, temporal validation, calibrated policy costs, adversarial testing, drift monitoring, privacy review, and human oversight.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/data/SMSSpamCollection
+5574
View File
File diff suppressed because it is too large Load Diff
Lab-4/Lab-4-Solution/data/sms+spam+collection.zip
BIN
View File
Binary file not shown.