{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ba1236aa",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 2: Bag of Words and TF-IDF\n",
    "\n",
    " \n",
    "**Topics:** CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66017ea",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- explain how Bag of Words maps documents to numeric vectors;\n",
    "- inspect a document-term matrix and its vocabulary;\n",
    "- fit TF-IDF on training text and transform unseen text;\n",
    "- describe why text matrices are high-dimensional and sparse;\n",
    "- compare unigram features with unigram-plus-bigram features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4a39b50",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5850d02",
   "metadata": {},
   "source": [
    "## 1. Make Bag of Words visible\n",
    "\n",
    "Use three short messages so that every row, token, and count can be inspected directly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c21fbff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tiny_messages = [\n",
    "    \"free prize now\",\n",
    "    \"free meeting today\",\n",
    "    \"project meeting today\",\n",
    "]\n",
    "\n",
    "# TODO 1: fit a CountVectorizer on tiny_messages and transform the messages.\n",
    "count_vectorizer = None\n",
    "tiny_bow = None\n",
    "\n",
    "# TODO 2: create a DataFrame whose columns are the learned feature names.\n",
    "tiny_bow_df = None\n",
    "tiny_bow_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92098ba9",
   "metadata": {},
   "source": [
    "## 2. Prepare the UCI messages\n",
    "\n",
    "Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0fc369e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
    "df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
    "df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
    "df = df.dropna(subset=[\"label\", \"message\"])\n",
    "df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
    "\n",
    "conflicts = df.groupby(\"message\")[\"label\"].nunique()\n",
    "conflict_texts = conflicts[conflicts.gt(1)].index\n",
    "df = df[~df[\"message\"].isin(conflict_texts)]\n",
    "df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
    "\n",
    "X = df[\"message\"]\n",
    "y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
    "print(\"Clean messages:\", len(df))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c01ce3be",
   "metadata": {},
   "source": [
    "## 3. Fit TF-IDF without leakage\n",
    "\n",
    "`fit_transform` learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only `transform`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12627cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: split message and target into 80% training and 20% test data.\n",
    "# Use random_state=42 and stratify=y.\n",
    "X_train, X_test, y_train, y_test = None, None, None, None\n",
    "\n",
    "# TODO 4: fit TfidfVectorizer on X_train only and transform X_test.\n",
    "tfidf_vectorizer = None\n",
    "X_train_tfidf = None\n",
    "X_test_tfidf = None\n",
    "\n",
    "print(\"Training matrix shape:\", None)\n",
    "print(\"Test matrix shape:\", None)\n",
    "print(\"Training density:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c6ee2c2",
   "metadata": {},
   "source": [
    "## 4. Inspect the learned feature space\n",
    "\n",
    "A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42745259",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: calculate each term's mean TF-IDF weight across training messages.\n",
    "mean_tfidf = None\n",
    "\n",
    "# TODO 6: build a descending Series and keep the 15 highest-weight terms.\n",
    "top_terms = None\n",
    "top_terms\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d60a97ed",
   "metadata": {},
   "source": [
    "## 5. Add short phrases with bigrams\n",
    "\n",
    "A unigram represents one token. A bigram can preserve short phrases such as `call now` or `free entry`, but it also increases the number of features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b6458f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: fit a second TF-IDF vectorizer with unigrams and bigrams.\n",
    "ngram_vectorizer = None\n",
    "X_train_ngram = None\n",
    "\n",
    "print(\"Unigram features:\", None)\n",
    "print(\"Unigram + bigram features:\", None)\n",
    "print(\"Example bigrams:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5703a0a9",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46da5f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))\n",
    "assert tiny_bow_df.loc[\"message_1\", \"free\"] == 1\n",
    "assert len(X_train) + len(X_test) == len(df)\n",
    "assert X_train_tfidf.shape[0] == len(X_train)\n",
    "assert X_test_tfidf.shape[0] == len(X_test)\n",
    "assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]\n",
    "assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]\n",
    "assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]\n",
    "assert len(top_terms) == 15\n",
    "print(\"✓ Lab 2 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31bb9a13",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why is a text feature matrix usually sparse?\n",
    "2. Why must the vectorizer be fitted only on training messages?\n",
    "3. What trade-off is introduced by adding bigrams?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}