labs/Lab-6/Lab-6-Practical/02_Flow_Features_Preprocessing_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
    "\n",
    "\n",
    "**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- interpret basic, content, traffic, and host-based flow features;\n",
    "- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
    "- compare normal and attack distributions;\n",
    "- build separate preprocessing plans for distance models and tree models;\n",
    "- fit encoders and scalers using training data only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
    "TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
    "\n",
    "missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
    "if missing_files:\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
    "        \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
    "    )\n",
    "\n",
    "TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
    "CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
    "\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load both splits and define feature_cols/numeric_features.\n",
    "train_df = None\n",
    "test_df = None\n",
    "feature_cols = None\n",
    "numeric_features = None\n",
    "\n",
    "# TODO 2: create feature_families for basic, content, traffic, and host-based features.\n",
    "feature_families = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Inspect byte-count skew and apply `log1p`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 3: create log_src_bytes and log_dst_bytes using np.log1p.\n",
    "exploratory = None\n",
    "\n",
    "# TODO 4: calculate class medians and plot a sampled log-byte scatter chart.\n",
    "byte_summary = None\n",
    "sample = None\n",
    "display(byte_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Build preprocessing plans for two model families"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: build supervised_preprocessor with OneHotEncoder for the three\n",
    "# categorical features and median imputation for numeric features.\n",
    "supervised_preprocessor = None\n",
    "\n",
    "distance_features = [\n",
    "    \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
    "    \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
    "]\n",
    "# TODO 6: build distance_preprocessor with median imputation and StandardScaler.\n",
    "distance_preprocessor = None\n",
    "\n",
    "# TODO 7: fit both preprocessors on training data only and record output shapes.\n",
    "supervised_shape = None\n",
    "distance_shape = None\n",
    "print(supervised_shape, distance_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## Why two plans?\n",
    "\n",
    "Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-011",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(feature_cols) == 41\n",
    "assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
    "assert supervised_shape[0] == 100\n",
    "assert supervised_shape[1] > len(feature_cols)\n",
    "assert distance_shape == (100, len(distance_features))\n",
    "assert all(feature in numeric_features for feature in distance_features)\n",
    "print(\"✓ Lab 2 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why is `log1p` preferable to dropping zero-byte flows?\n",
    "2. Why do distance methods need scaling while Decision Trees usually do not?\n",
    "3. What information leaks when a scaler or encoder is fitted on test data?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}