{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d1c571ce",
   "metadata": {},
   "source": [
    "# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
    "\n",
    "\n",
    "**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
    "\n",
    "> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "209fc458",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- remove duplicate records and normalize invalid numeric values;\n",
    "- create a stratified train-test split;\n",
    "- fit imputation and scaling operations on training data only;\n",
    "- explain how fitting preprocessing on the test set causes leakage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "660ee68c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a03355",
   "metadata": {},
   "source": [
    "## 1. Load and clean the raw table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8962439b",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df = pd.read_csv(DATA_PATH)\n",
    "\n",
    "# TODO 1: remove surrounding whitespace from every column name\n",
    "raw_df.columns = None\n",
    "\n",
    "# TODO 2: remove duplicate rows and store the result in clean_df\n",
    "clean_df = None\n",
    "\n",
    "# TODO 3: replace positive and negative infinity with NaN\n",
    "clean_df = None\n",
    "\n",
    "print(\"Raw rows:\", len(raw_df))\n",
    "print(\"Clean rows:\", None)\n",
    "print(\"Missing numeric values:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840a6b05",
   "metadata": {},
   "source": [
    "## 2. Create features, target, and a stratified split\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bd85328",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 4: select numeric features and create a binary target\n",
    "X = None\n",
    "y = None\n",
    "\n",
    "# TODO 5: create an 80/20 split with random_state=42 and stratify=y\n",
    "X_train, X_test, y_train, y_test = None, None, None, None\n",
    "\n",
    "print(\"Train shape:\", None)\n",
    "print(\"Test shape:\", None)\n",
    "print(\"Train attack rate:\", None)\n",
    "print(\"Test attack rate:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0408dc06",
   "metadata": {},
   "source": [
    "## 3. Fit preprocessing on training data only\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88ffbc3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer = SimpleImputer(strategy=\"median\")\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# TODO 6: fit the imputer on X_train, then transform train and test\n",
    "X_train_imputed = None\n",
    "X_test_imputed = None\n",
    "\n",
    "# TODO 7: fit the scaler on imputed training data, then transform train and test\n",
    "X_train_scaled = None\n",
    "X_test_scaled = None\n",
    "\n",
    "print(\"Mean of scaled training columns:\", None)\n",
    "print(\"Mean of scaled test columns:\", None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "578c5e87",
   "metadata": {},
   "source": [
    "## Why the test mean is not exactly zero\n",
    "\n",
    "`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "639b3502",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b00a51c",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(clean_df) < len(raw_df)\n",
    "assert np.isfinite(X_train_scaled).all()\n",
    "assert np.isfinite(X_test_scaled).all()\n",
    "assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
    "assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
    "assert X_train_scaled.shape[1] == X.shape[1]\n",
    "print(\"✓ Lab 2 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0461854",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why must the imputer learn medians from the training set only?\n",
    "2. What information would leak if the scaler were fit before the train-test split?\n",
    "3. Why is stratification useful for an imbalanced security target?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}