CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Lab-3/Lab-3-Practical/02_Clean_Split_No_Leakage_Lab.ipynb

@@ -0,0 +1,207 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d1c571ce",
+   "metadata": {},
+   "source": [
+    "# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
+    "\n",
+    "\n",
+    "**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
+    "\n",
+    "> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "209fc458",
+   "metadata": {},
+   "source": [
+    "## Learning objectives\n",
+    "\n",
+    "After completing this lab, you will be able to:\n",
+    "\n",
+    "- remove duplicate records and normalize invalid numeric values;\n",
+    "- create a stratified train-test split;\n",
+    "- fit imputation and scaling operations on training data only;\n",
+    "- explain how fitting preprocessing on the test set causes leakage.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "660ee68c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
+    "RANDOM_STATE = 42\n",
+    "\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.impute import SimpleImputer\n",
+    "from sklearn.preprocessing import StandardScaler\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2a03355",
+   "metadata": {},
+   "source": [
+    "## 1. Load and clean the raw table\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8962439b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "raw_df = pd.read_csv(DATA_PATH)\n",
+    "\n",
+    "# TODO 1: remove surrounding whitespace from every column name\n",
+    "raw_df.columns = None\n",
+    "\n",
+    "# TODO 2: remove duplicate rows and store the result in clean_df\n",
+    "clean_df = None\n",
+    "\n",
+    "# TODO 3: replace positive and negative infinity with NaN\n",
+    "clean_df = None\n",
+    "\n",
+    "print(\"Raw rows:\", len(raw_df))\n",
+    "print(\"Clean rows:\", None)\n",
+    "print(\"Missing numeric values:\", None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "840a6b05",
+   "metadata": {},
+   "source": [
+    "## 2. Create features, target, and a stratified split\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4bd85328",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# TODO 4: select numeric features and create a binary target\n",
+    "X = None\n",
+    "y = None\n",
+    "\n",
+    "# TODO 5: create an 80/20 split with random_state=42 and stratify=y\n",
+    "X_train, X_test, y_train, y_test = None, None, None, None\n",
+    "\n",
+    "print(\"Train shape:\", None)\n",
+    "print(\"Test shape:\", None)\n",
+    "print(\"Train attack rate:\", None)\n",
+    "print(\"Test attack rate:\", None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0408dc06",
+   "metadata": {},
+   "source": [
+    "## 3. Fit preprocessing on training data only\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "88ffbc3e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "imputer = SimpleImputer(strategy=\"median\")\n",
+    "scaler = StandardScaler()\n",
+    "\n",
+    "# TODO 6: fit the imputer on X_train, then transform train and test\n",
+    "X_train_imputed = None\n",
+    "X_test_imputed = None\n",
+    "\n",
+    "# TODO 7: fit the scaler on imputed training data, then transform train and test\n",
+    "X_train_scaled = None\n",
+    "X_test_scaled = None\n",
+    "\n",
+    "print(\"Mean of scaled training columns:\", None)\n",
+    "print(\"Mean of scaled test columns:\", None)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "578c5e87",
+   "metadata": {},
+   "source": [
+    "## Why the test mean is not exactly zero\n",
+    "\n",
+    "`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "639b3502",
+   "metadata": {},
+   "source": [
+    "## Self-check\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1b00a51c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert len(clean_df) < len(raw_df)\n",
+    "assert np.isfinite(X_train_scaled).all()\n",
+    "assert np.isfinite(X_test_scaled).all()\n",
+    "assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
+    "assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
+    "assert X_train_scaled.shape[1] == X.shape[1]\n",
+    "print(\"✓ Lab 2 completed\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e0461854",
+   "metadata": {},
+   "source": [
+    "## Reflection questions\n",
+    "\n",
+    "1. Why must the imputer learn medians from the training set only?\n",
+    "2. What information would leak if the scaler were fit before the train-test split?\n",
+    "3. Why is stratification useful for an imbalanced security target?\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}