labs/Lab-4/Lab-4-Practical/01_SMS_Dataset_Inspection_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1545411a",
   "metadata": {},
   "source": [
    "# Module 4 — Lab 1: Inspecting the SMS Spam Dataset\n",
    "\n",
    " \n",
    "**Topics:** text data, duplicates, conflicting labels, class distribution\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac8bdc2a",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- load the original UCI tab-separated text file;\n",
    "- audit missing values, exact duplicates, and conflicting labels;\n",
    "- create a binary target without modifying the message text;\n",
    "- describe class imbalance and message-length differences;\n",
    "- explain why aggressive cleaning may remove useful security signals.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c903f90d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "\n",
    "DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/SMSSpamCollection. Follow data/README.md to download \"\n",
    "        \"and extract the official UCI SMS Spam Collection.\"\n",
    "    )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc48c80",
   "metadata": {},
   "source": [
    "## 1. Load the original collection\n",
    "\n",
    "The source file has no header. Each row contains a label, one tab character, and the message text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcba4fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load the tab-separated UCI file into raw_df.\n",
    "# Use the column names label and message.\n",
    "raw_df = None\n",
    "\n",
    "# TODO 2: display the shape, first five rows, and raw label counts.\n",
    "print(\"Raw shape:\", None)\n",
    "raw_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585072d3",
   "metadata": {},
   "source": [
    "## 2. Audit the records\n",
    "\n",
    "Exact duplicates can cross a random train-test split and make evaluation too optimistic. A more serious problem occurs when identical text has different labels. Audit both conditions before modeling.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72d9609c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: normalize label whitespace and remove rows with missing or empty text.\n",
    "audited_df = None\n",
    "\n",
    "# TODO 4: count exact duplicate rows.\n",
    "exact_duplicate_count = None\n",
    "\n",
    "# TODO 5: find messages associated with more than one distinct label.\n",
    "conflicting_messages = None\n",
    "\n",
    "print(\"Exact duplicate rows:\", exact_duplicate_count)\n",
    "print(\"Conflicting message texts:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e3d7d2",
   "metadata": {},
   "source": [
    "## 3. Build the clean classroom table\n",
    "\n",
    "Remove ambiguous messages before exact duplicates, then map `ham → 0` and `spam → 1`. Keep the original text intact.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bf6d47d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 6: remove every conflicting message and then remove exact duplicates.\n",
    "clean_df = None\n",
    "\n",
    "# TODO 7: map ham to 0 and spam to 1 in a new target column.\n",
    "clean_df[\"target\"] = None\n",
    "\n",
    "print(\"Clean shape:\", None)\n",
    "print(clean_df[\"target\"].value_counts().sort_index())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abd58b67",
   "metadata": {},
   "source": [
    "## 4. Describe the classes and message lengths\n",
    "\n",
    "Length is used only for exploration. The later models will learn from the message text itself.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ae01028",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 8: create character_count and word_count columns.\n",
    "clean_df[\"character_count\"] = None\n",
    "clean_df[\"word_count\"] = None\n",
    "\n",
    "# TODO 9: calculate a summary indexed by label with count and median lengths.\n",
    "length_summary = None\n",
    "class_distribution = None\n",
    "\n",
    "print(length_summary)\n",
    "print(\"Class proportions:\")\n",
    "print(class_distribution)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca6f77a2",
   "metadata": {},
   "source": [
    "## Security note: do not over-clean\n",
    "\n",
    "URLs, phone numbers, currency symbols, punctuation, unusual capitalization, and deliberate misspellings may contain spam evidence. Begin with the vectorizer defaults. Add normalization only when an experiment demonstrates that it improves validation performance without hiding operationally important signals.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9559352e",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb0dabd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert list(raw_df.columns) == [\"label\", \"message\"]\n",
    "assert clean_df[\"message\"].notna().all()\n",
    "assert clean_df[\"message\"].ne(\"\").all()\n",
    "assert set(clean_df[\"label\"].unique()) == {\"ham\", \"spam\"}\n",
    "assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
    "assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
    "assert not clean_df[\"message\"].isin(conflicting_messages).any()\n",
    "assert class_distribution.loc[\"ham\"] > class_distribution.loc[\"spam\"]\n",
    "assert {\"character_count\", \"word_count\"}.issubset(clean_df.columns)\n",
    "print(\"✓ Lab 1 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62472b1f",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why should exact duplicates be removed before the train-test split?\n",
    "2. Why might removing URLs and currency symbols reduce detection quality?\n",
    "3. Why is accuracy alone risky for this dataset?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}