CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,273 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "942ac4eb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 4 — Lab 2: Bag of Words and TF-IDF\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "554dc6ef",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- explain how Bag of Words maps documents to numeric vectors;\n",
|
||||
"- inspect a document-term matrix and its vocabulary;\n",
|
||||
"- fit TF-IDF on training text and transform unseen text;\n",
|
||||
"- describe why text matrices are high-dimensional and sparse;\n",
|
||||
"- compare unigram features with unigram-plus-bigram features.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dee543fe",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
|
||||
"RANDOM_STATE = 42\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
|
||||
" )\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "637b0eaa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Make Bag of Words visible\n",
|
||||
"\n",
|
||||
"Use three short messages so that every row, token, and count can be inspected directly.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3c3d5152",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tiny_messages = [\n",
|
||||
" \"free prize now\",\n",
|
||||
" \"free meeting today\",\n",
|
||||
" \"project meeting today\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"count_vectorizer = CountVectorizer()\n",
|
||||
"tiny_bow = count_vectorizer.fit_transform(tiny_messages)\n",
|
||||
"tiny_bow_df = pd.DataFrame(\n",
|
||||
" tiny_bow.toarray(),\n",
|
||||
" columns=count_vectorizer.get_feature_names_out(),\n",
|
||||
" index=[\"message_1\", \"message_2\", \"message_3\"],\n",
|
||||
")\n",
|
||||
"tiny_bow_df\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e3e98734",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Prepare the UCI messages\n",
|
||||
"\n",
|
||||
"Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9841f9b7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
|
||||
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
|
||||
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
|
||||
"df = df.dropna(subset=[\"label\", \"message\"])\n",
|
||||
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
|
||||
"\n",
|
||||
"conflicts = df.groupby(\"message\")[\"label\"].nunique()\n",
|
||||
"conflict_texts = conflicts[conflicts.gt(1)].index\n",
|
||||
"df = df[~df[\"message\"].isin(conflict_texts)]\n",
|
||||
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
|
||||
"\n",
|
||||
"X = df[\"message\"]\n",
|
||||
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
|
||||
"print(\"Clean messages:\", len(df))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "92848cea",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Fit TF-IDF without leakage\n",
|
||||
"\n",
|
||||
"`fit_transform` learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only `transform`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e5a89189",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X,\n",
|
||||
" y,\n",
|
||||
" test_size=0.20,\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" stratify=y,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"tfidf_vectorizer = TfidfVectorizer(lowercase=True)\n",
|
||||
"X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)\n",
|
||||
"X_test_tfidf = tfidf_vectorizer.transform(X_test)\n",
|
||||
"\n",
|
||||
"density = X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1])\n",
|
||||
"print(\"Training matrix shape:\", X_train_tfidf.shape)\n",
|
||||
"print(\"Test matrix shape:\", X_test_tfidf.shape)\n",
|
||||
"print(\"Training density:\", f\"{density:.4%}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a4230176",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Inspect the learned feature space\n",
|
||||
"\n",
|
||||
"A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "90e679d0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"mean_tfidf = np.asarray(X_train_tfidf.mean(axis=0)).ravel()\n",
|
||||
"top_terms = pd.Series(\n",
|
||||
" mean_tfidf,\n",
|
||||
" index=tfidf_vectorizer.get_feature_names_out(),\n",
|
||||
" name=\"mean_tfidf\",\n",
|
||||
").sort_values(ascending=False).head(15)\n",
|
||||
"top_terms\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ebe7eee",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. Add short phrases with bigrams\n",
|
||||
"\n",
|
||||
"A unigram represents one token. A bigram can preserve short phrases such as `call now` or `free entry`, but it also increases the number of features.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6c1de19b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ngram_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1)\n",
|
||||
"X_train_ngram = ngram_vectorizer.fit_transform(X_train)\n",
|
||||
"ngram_features = ngram_vectorizer.get_feature_names_out()\n",
|
||||
"example_bigrams = [term for term in ngram_features if \" \" in term][:10]\n",
|
||||
"\n",
|
||||
"print(\"Unigram features:\", X_train_tfidf.shape[1])\n",
|
||||
"print(\"Unigram + bigram features:\", X_train_ngram.shape[1])\n",
|
||||
"print(\"Example bigrams:\", example_bigrams)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "64432265",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "36f18994",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))\n",
|
||||
"assert tiny_bow_df.loc[\"message_1\", \"free\"] == 1\n",
|
||||
"assert len(X_train) + len(X_test) == len(df)\n",
|
||||
"assert X_train_tfidf.shape[0] == len(X_train)\n",
|
||||
"assert X_test_tfidf.shape[0] == len(X_test)\n",
|
||||
"assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]\n",
|
||||
"assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]\n",
|
||||
"assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]\n",
|
||||
"assert len(top_terms) == 15\n",
|
||||
"print(\"✓ Lab 2 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "991fa410",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Suggested answers\n",
|
||||
"\n",
|
||||
"1. **Why is a text feature matrix usually sparse?** \n",
|
||||
" Each message uses only a small fraction of the complete vocabulary, so most document-term cells are zero.\n",
|
||||
"2. **Why must the vectorizer be fitted only on training messages?** \n",
|
||||
" Fitting on all messages reveals test-set vocabulary and frequency information during training, which is data leakage and can make evaluation optimistic.\n",
|
||||
"3. **What trade-off is introduced by adding bigrams?** \n",
|
||||
" Bigrams preserve useful short phrases but expand the feature space, consume more memory, and can overfit when phrases are rare.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user