labs/Lab-6/Lab-6-Solution/02_Flow_Features_Preprocessing_Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
    "\n",
    "\n",
    "**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
    "\n",
    "> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- interpret basic, content, traffic, and host-based flow features;\n",
    "- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
    "- compare normal and attack distributions;\n",
    "- build separate preprocessing plans for distance models and tree models;\n",
    "- fit encoders and scalers using training data only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
    "TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
    "\n",
    "missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
    "if missing_files:\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
    "        \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
    "    )\n",
    "\n",
    "TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
    "CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
    "\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
    "test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
    "feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
    "numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
    "\n",
    "feature_families = {\n",
    "    \"basic\": [\"duration\", \"protocol_type\", \"service\", \"flag\", \"src_bytes\", \"dst_bytes\", \"land\", \"wrong_fragment\", \"urgent\"],\n",
    "    \"content\": numeric_features[6:19],\n",
    "    \"traffic\": numeric_features[19:28],\n",
    "    \"host_based\": numeric_features[28:],\n",
    "}\n",
    "{family: len(names) for family, names in feature_families.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Inspect byte-count skew and apply `log1p`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "exploratory = train_df[[\"src_bytes\", \"dst_bytes\", \"binary_label\"]].copy()\n",
    "exploratory[\"log_src_bytes\"] = np.log1p(exploratory[\"src_bytes\"])\n",
    "exploratory[\"log_dst_bytes\"] = np.log1p(exploratory[\"dst_bytes\"])\n",
    "\n",
    "byte_summary = exploratory.groupby(\"binary_label\")[[\n",
    "    \"src_bytes\", \"dst_bytes\", \"log_src_bytes\", \"log_dst_bytes\"\n",
    "]].median()\n",
    "display(byte_summary)\n",
    "\n",
    "sample = exploratory.sample(min(5000, len(exploratory)), random_state=42)\n",
    "colors = sample[\"binary_label\"].map({0: \"#4C78A8\", 1: \"#E45756\"})\n",
    "ax = sample.plot.scatter(\n",
    "    x=\"log_src_bytes\", y=\"log_dst_bytes\", c=colors, alpha=0.25, figsize=(8, 6)\n",
    ")\n",
    "ax.set_title(\"NSL-KDD flows after log1p byte transformation\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Build preprocessing plans for two model families"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "supervised_preprocessor = ColumnTransformer([\n",
    "    (\"categorical\", OneHotEncoder(handle_unknown=\"ignore\"), CATEGORICAL_FEATURES),\n",
    "    (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
    "])\n",
    "\n",
    "distance_features = [\n",
    "    \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
    "    \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
    "]\n",
    "distance_preprocessor = Pipeline([\n",
    "    (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
    "    (\"scaler\", StandardScaler()),\n",
    "])\n",
    "\n",
    "supervised_preprocessor.fit(train_df[feature_cols])\n",
    "distance_preprocessor.fit(train_df[distance_features])\n",
    "supervised_shape = supervised_preprocessor.transform(train_df[feature_cols].head(100)).shape\n",
    "distance_shape = distance_preprocessor.transform(train_df[distance_features].head(100)).shape\n",
    "print(\"Supervised transformed shape:\", supervised_shape)\n",
    "print(\"Distance transformed shape:\", distance_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## Why two plans?\n",
    "\n",
    "Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-011",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(feature_cols) == 41\n",
    "assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
    "assert supervised_shape[0] == 100\n",
    "assert supervised_shape[1] > len(feature_cols)\n",
    "assert distance_shape == (100, len(distance_features))\n",
    "assert all(feature in numeric_features for feature in distance_features)\n",
    "print(\"✓ Lab 2 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Suggested answers\n",
    "\n",
    "1. **Why `log1p`?** It compresses large byte values and remains defined at zero.\n",
    "2. **Why scale distance features?** Otherwise byte counts can dominate rate features in Euclidean distance.\n",
    "3. **Why fit on training only?** Test categories and distribution statistics must not influence the learned preprocessing state."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}