CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
@@ -0,0 +1,224 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-001",
+   "metadata": {},
+   "source": [
+    "# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
+    "\n",
+    "\n",
+    "**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
+    "\n",
+    "> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-002",
+   "metadata": {},
+   "source": [
+    "## Learning objectives\n",
+    "\n",
+    "After completing this lab, you will be able to:\n",
+    "\n",
+    "- interpret basic, content, traffic, and host-based flow features;\n",
+    "- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
+    "- compare normal and attack distributions;\n",
+    "- build separate preprocessing plans for distance models and tree models;\n",
+    "- fit encoders and scalers using training data only."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-003",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
+    "TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
+    "\n",
+    "missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
+    "if missing_files:\n",
+    "    raise FileNotFoundError(\n",
+    "        \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
+    "        \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
+    "    )\n",
+    "\n",
+    "TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
+    "CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
+    "\n",
+    "\n",
+    "from sklearn.compose import ColumnTransformer\n",
+    "from sklearn.impute import SimpleImputer\n",
+    "from sklearn.pipeline import Pipeline\n",
+    "from sklearn.preprocessing import OneHotEncoder, StandardScaler"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-004",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
+    "test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
+    "feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
+    "numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
+    "\n",
+    "feature_families = {\n",
+    "    \"basic\": [\"duration\", \"protocol_type\", \"service\", \"flag\", \"src_bytes\", \"dst_bytes\", \"land\", \"wrong_fragment\", \"urgent\"],\n",
+    "    \"content\": numeric_features[6:19],\n",
+    "    \"traffic\": numeric_features[19:28],\n",
+    "    \"host_based\": numeric_features[28:],\n",
+    "}\n",
+    "{family: len(names) for family, names in feature_families.items()}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-005",
+   "metadata": {},
+   "source": [
+    "## 1. Inspect byte-count skew and apply `log1p`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-006",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "exploratory = train_df[[\"src_bytes\", \"dst_bytes\", \"binary_label\"]].copy()\n",
+    "exploratory[\"log_src_bytes\"] = np.log1p(exploratory[\"src_bytes\"])\n",
+    "exploratory[\"log_dst_bytes\"] = np.log1p(exploratory[\"dst_bytes\"])\n",
+    "\n",
+    "byte_summary = exploratory.groupby(\"binary_label\")[[\n",
+    "    \"src_bytes\", \"dst_bytes\", \"log_src_bytes\", \"log_dst_bytes\"\n",
+    "]].median()\n",
+    "display(byte_summary)\n",
+    "\n",
+    "sample = exploratory.sample(min(5000, len(exploratory)), random_state=42)\n",
+    "colors = sample[\"binary_label\"].map({0: \"#4C78A8\", 1: \"#E45756\"})\n",
+    "ax = sample.plot.scatter(\n",
+    "    x=\"log_src_bytes\", y=\"log_dst_bytes\", c=colors, alpha=0.25, figsize=(8, 6)\n",
+    ")\n",
+    "ax.set_title(\"NSL-KDD flows after log1p byte transformation\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-007",
+   "metadata": {},
+   "source": [
+    "## 2. Build preprocessing plans for two model families"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-008",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "supervised_preprocessor = ColumnTransformer([\n",
+    "    (\"categorical\", OneHotEncoder(handle_unknown=\"ignore\"), CATEGORICAL_FEATURES),\n",
+    "    (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
+    "])\n",
+    "\n",
+    "distance_features = [\n",
+    "    \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
+    "    \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
+    "]\n",
+    "distance_preprocessor = Pipeline([\n",
+    "    (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
+    "    (\"scaler\", StandardScaler()),\n",
+    "])\n",
+    "\n",
+    "supervised_preprocessor.fit(train_df[feature_cols])\n",
+    "distance_preprocessor.fit(train_df[distance_features])\n",
+    "supervised_shape = supervised_preprocessor.transform(train_df[feature_cols].head(100)).shape\n",
+    "distance_shape = distance_preprocessor.transform(train_df[distance_features].head(100)).shape\n",
+    "print(\"Supervised transformed shape:\", supervised_shape)\n",
+    "print(\"Distance transformed shape:\", distance_shape)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-009",
+   "metadata": {},
+   "source": [
+    "## Why two plans?\n",
+    "\n",
+    "Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-010",
+   "metadata": {},
+   "source": [
+    "## Self-check"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-011",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert len(feature_cols) == 41\n",
+    "assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
+    "assert supervised_shape[0] == 100\n",
+    "assert supervised_shape[1] > len(feature_cols)\n",
+    "assert distance_shape == (100, len(distance_features))\n",
+    "assert all(feature in numeric_features for feature in distance_features)\n",
+    "print(\"✓ Lab 2 completed\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-012",
+   "metadata": {},
+   "source": [
+    "## Suggested answers\n",
+    "\n",
+    "1. **Why `log1p`?** It compresses large byte values and remains defined at zero.\n",
+    "2. **Why scale distance features?** Otherwise byte counts can dominate rate features in Euclidean distance.\n",
+    "3. **Why fit on training only?** Test categories and distribution statistics must not influence the learned preprocessing state."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}