{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8aacddc0",
   "metadata": {},
   "source": [
    "# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
    "\n",
    "\n",
    "**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
    "\n",
    "> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26e6221",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- remove duplicate records and normalize invalid numeric values;\n",
    "- create a stratified train-test split;\n",
    "- fit imputation and scaling operations on training data only;\n",
    "- explain how fitting preprocessing on the test set causes leakage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e440f789",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28421713",
   "metadata": {},
   "source": [
    "## 1. Load and clean the raw table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "797874c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df = pd.read_csv(DATA_PATH)\n",
    "raw_df.columns = raw_df.columns.str.strip()\n",
    "clean_df = raw_df.drop_duplicates().copy()\n",
    "clean_df = clean_df.replace([np.inf, -np.inf], np.nan)\n",
    "\n",
    "print(\"Raw rows:\", len(raw_df))\n",
    "print(\"Clean rows:\", len(clean_df))\n",
    "print(\"Missing numeric values:\", int(clean_df.select_dtypes(include=\"number\").isna().sum().sum()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f8834a",
   "metadata": {},
   "source": [
    "## 2. Create features, target, and a stratified split\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b0b8cf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = clean_df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
    "y = clean_df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X,\n",
    "    y,\n",
    "    test_size=0.2,\n",
    "    random_state=RANDOM_STATE,\n",
    "    stratify=y,\n",
    ")\n",
    "\n",
    "print(\"Train shape:\", X_train.shape)\n",
    "print(\"Test shape:\", X_test.shape)\n",
    "print(\"Train attack rate:\", round(y_train.mean(), 3))\n",
    "print(\"Test attack rate:\", round(y_test.mean(), 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a742c62a",
   "metadata": {},
   "source": [
    "## 3. Fit preprocessing on training data only\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43a2c168",
   "metadata": {},
   "outputs": [],
   "source": [
    "imputer = SimpleImputer(strategy=\"median\")\n",
    "scaler = StandardScaler()\n",
    "\n",
    "X_train_imputed = imputer.fit_transform(X_train)\n",
    "X_test_imputed = imputer.transform(X_test)\n",
    "\n",
    "X_train_scaled = scaler.fit_transform(X_train_imputed)\n",
    "X_test_scaled = scaler.transform(X_test_imputed)\n",
    "\n",
    "print(\"Mean of scaled training columns:\", np.round(X_train_scaled.mean(axis=0), 3))\n",
    "print(\"Mean of scaled test columns:\", np.round(X_test_scaled.mean(axis=0), 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c3fc6ca",
   "metadata": {},
   "source": [
    "## Why the test mean is not exactly zero\n",
    "\n",
    "`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459304bb",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "478a41eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(clean_df) <= len(raw_df)\n",
    "assert np.isfinite(X_train_scaled).all()\n",
    "assert np.isfinite(X_test_scaled).all()\n",
    "assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
    "assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
    "assert X_train_scaled.shape[1] == X.shape[1]\n",
    "print(\"✓ Lab 2 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca45be08",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why must the imputer learn medians from the training set only?\n",
    "2. What information would leak if the scaler were fit before the train-test split?\n",
    "3. Why is stratification useful for an imbalanced security target?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}