mikrotik/labs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Browse Source
This commit is contained in:
mikrotik
2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Lab-6/Lab-6-Solution/01_NSL_KDD_Flow_Audit_Solution.ipynb
+709
View File
@@ -0,0 +1,709 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 1: Auditing NSL-KDD Flow Records\n",
"\n",
"\n",
"**Topics:** network flows, fixed split, labels, attack groups, leakage\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- distinguish network-flow features from targets and audit fields;\n",
"- preserve the predefined NSL-KDD train/test split;\n",
"- describe binary labels and the DoS, Probe, R2L, and U2R groups;\n",
"- identify attack labels that appear only in the test set;\n",
"- document imbalance, duplicates, and benchmark limitations."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load and validate the fixed split"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cell-005",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Train shape: (125973, 45)\n",
"Test shape: (22544, 45)\n",
"Model features: 41\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>duration</th>\n",
" <th>protocol_type</th>\n",
" <th>service</th>\n",
" <th>flag</th>\n",
" <th>src_bytes</th>\n",
" <th>dst_bytes</th>\n",
" <th>land</th>\n",
" <th>wrong_fragment</th>\n",
" <th>urgent</th>\n",
" <th>hot</th>\n",
" <th>...</th>\n",
" <th>dst_host_same_src_port_rate</th>\n",
" <th>dst_host_srv_diff_host_rate</th>\n",
" <th>dst_host_serror_rate</th>\n",
" <th>dst_host_srv_serror_rate</th>\n",
" <th>dst_host_rerror_rate</th>\n",
" <th>dst_host_srv_rerror_rate</th>\n",
" <th>attack_label</th>\n",
" <th>attack_group</th>\n",
" <th>binary_label</th>\n",
" <th>difficulty</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>tcp</td>\n",
" <td>ftp_data</td>\n",
" <td>sf</td>\n",
" <td>491</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0.17</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.05</td>\n",
" <td>0.00</td>\n",
" <td>normal</td>\n",
" <td>Normal</td>\n",
" <td>0</td>\n",
" <td>20</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0</td>\n",
" <td>udp</td>\n",
" <td>other</td>\n",
" <td>sf</td>\n",
" <td>146</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0.88</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>normal</td>\n",
" <td>Normal</td>\n",
" <td>0</td>\n",
" <td>15</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0</td>\n",
" <td>tcp</td>\n",
" <td>private</td>\n",
" <td>s0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>1.00</td>\n",
" <td>1.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>neptune</td>\n",
" <td>DoS</td>\n",
" <td>1</td>\n",
" <td>19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0</td>\n",
" <td>tcp</td>\n",
" <td>http</td>\n",
" <td>sf</td>\n",
" <td>232</td>\n",
" <td>8153</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0.03</td>\n",
" <td>0.04</td>\n",
" <td>0.03</td>\n",
" <td>0.01</td>\n",
" <td>0.00</td>\n",
" <td>0.01</td>\n",
" <td>normal</td>\n",
" <td>Normal</td>\n",
" <td>0</td>\n",
" <td>21</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0</td>\n",
" <td>tcp</td>\n",
" <td>http</td>\n",
" <td>sf</td>\n",
" <td>199</td>\n",
" <td>420</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>0.00</td>\n",
" <td>normal</td>\n",
" <td>Normal</td>\n",
" <td>0</td>\n",
" <td>21</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 45 columns</p>\n",
"</div>"
],
"text/plain": [
" duration protocol_type service flag src_bytes dst_bytes land \\\n",
"0 0 tcp ftp_data sf 491 0 0 \n",
"1 0 udp other sf 146 0 0 \n",
"2 0 tcp private s0 0 0 0 \n",
"3 0 tcp http sf 232 8153 0 \n",
"4 0 tcp http sf 199 420 0 \n",
"\n",
" wrong_fragment urgent hot ... dst_host_same_src_port_rate \\\n",
"0 0 0 0 ... 0.17 \n",
"1 0 0 0 ... 0.88 \n",
"2 0 0 0 ... 0.00 \n",
"3 0 0 0 ... 0.03 \n",
"4 0 0 0 ... 0.00 \n",
"\n",
" dst_host_srv_diff_host_rate dst_host_serror_rate \\\n",
"0 0.00 0.00 \n",
"1 0.00 0.00 \n",
"2 0.00 1.00 \n",
"3 0.04 0.03 \n",
"4 0.00 0.00 \n",
"\n",
" dst_host_srv_serror_rate dst_host_rerror_rate dst_host_srv_rerror_rate \\\n",
"0 0.00 0.05 0.00 \n",
"1 0.00 0.00 0.00 \n",
"2 1.00 0.00 0.00 \n",
"3 0.01 0.00 0.01 \n",
"4 0.00 0.00 0.00 \n",
"\n",
" attack_label attack_group binary_label difficulty \n",
"0 normal Normal 0 20 \n",
"1 normal Normal 0 15 \n",
"2 neptune DoS 1 19 \n",
"3 normal Normal 0 21 \n",
"4 normal Normal 0 21 \n",
"\n",
"[5 rows x 45 columns]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
"test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
"feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
"numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
"\n",
"assert list(train_df.columns) == list(test_df.columns)\n",
"print(\"Train shape:\", train_df.shape)\n",
"print(\"Test shape:\", test_df.shape)\n",
"print(\"Model features:\", len(feature_cols))\n",
"display(train_df.head())"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit classes, groups, and unseen attack labels"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "cell-007",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>train</th>\n",
" <th>test</th>\n",
" </tr>\n",
" <tr>\n",
" <th>binary_label</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>67343</td>\n",
" <td>9711</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>58630</td>\n",
" <td>12833</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" train test\n",
"binary_label \n",
"0 67343 9711\n",
"1 58630 12833"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>train</th>\n",
" <th>test</th>\n",
" </tr>\n",
" <tr>\n",
" <th>attack_group</th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>DoS</th>\n",
" <td>45927</td>\n",
" <td>7460</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Normal</th>\n",
" <td>67343</td>\n",
" <td>9711</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Probe</th>\n",
" <td>11656</td>\n",
" <td>2421</td>\n",
" </tr>\n",
" <tr>\n",
" <th>R2L</th>\n",
" <td>995</td>\n",
" <td>2885</td>\n",
" </tr>\n",
" <tr>\n",
" <th>U2R</th>\n",
" <td>52</td>\n",
" <td>67</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" train test\n",
"attack_group \n",
"DoS 45927 7460\n",
"Normal 67343 9711\n",
"Probe 11656 2421\n",
"R2L 995 2885\n",
"U2R 52 67"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Attack labels seen only in test: ['apache2', 'httptunnel', 'mailbomb', 'mscan', 'named', 'processtable', 'ps', 'saint', 'sendmail', 'snmpgetattack', 'snmpguess', 'sqlattack', 'udpstorm', 'worm', 'xlock', 'xsnoop', 'xterm']\n"
]
}
],
"source": [
"train_binary = train_df[\"binary_label\"].value_counts().sort_index()\n",
"test_binary = test_df[\"binary_label\"].value_counts().sort_index()\n",
"train_groups = train_df[\"attack_group\"].value_counts()\n",
"test_groups = test_df[\"attack_group\"].value_counts()\n",
"\n",
"train_attack_labels = set(train_df.loc[train_df[\"binary_label\"] == 1, \"attack_label\"])\n",
"test_attack_labels = set(test_df.loc[test_df[\"binary_label\"] == 1, \"attack_label\"])\n",
"unseen_test_attacks = sorted(test_attack_labels - train_attack_labels)\n",
"\n",
"display(pd.DataFrame({\"train\": train_binary, \"test\": test_binary}).fillna(0).astype(int))\n",
"display(pd.DataFrame({\"train\": train_groups, \"test\": test_groups}).fillna(0).astype(int))\n",
"print(\"Attack labels seen only in test:\", unseen_test_attacks)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit quality and leakage risks"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cell-009",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'train_exact_duplicates': 0, 'test_exact_duplicates': 0}\n",
"Columns excluded from modeling: ['attack_group', 'attack_label', 'binary_label', 'difficulty']\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>train_missing</th>\n",
" <th>test_missing</th>\n",
" <th>train_unique</th>\n",
" <th>test_unique</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>num_outbound_cmds</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>is_host_login</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>is_guest_login</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>land</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>logged_in</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>root_shell</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>protocol_type</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>wrong_fragment</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>num_shells</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>su_attempted</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" train_missing test_missing train_unique test_unique\n",
"num_outbound_cmds 0 0 1 1\n",
"is_host_login 0 0 2 2\n",
"is_guest_login 0 0 2 2\n",
"land 0 0 2 2\n",
"logged_in 0 0 2 2\n",
"root_shell 0 0 2 2\n",
"protocol_type 0 0 3 3\n",
"wrong_fragment 0 0 3 3\n",
"num_shells 0 0 3 4\n",
"su_attempted 0 0 3 3"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"quality_summary = pd.DataFrame({\n",
" \"train_missing\": train_df[feature_cols].isna().sum(),\n",
" \"test_missing\": test_df[feature_cols].isna().sum(),\n",
" \"train_unique\": train_df[feature_cols].nunique(dropna=True),\n",
" \"test_unique\": test_df[feature_cols].nunique(dropna=True),\n",
"})\n",
"duplicate_counts = {\n",
" \"train_exact_duplicates\": int(train_df.duplicated().sum()),\n",
" \"test_exact_duplicates\": int(test_df.duplicated().sum()),\n",
"}\n",
"forbidden_model_columns = sorted(set(TARGET_COLUMNS) & set(train_df.columns))\n",
"\n",
"print(duplicate_counts)\n",
"print(\"Columns excluded from modeling:\", forbidden_model_columns)\n",
"display(quality_summary.sort_values(\"train_unique\").head(10))"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Benchmark note\n",
"\n",
"NSL-KDD is useful for learning a reproducible IDS workflow. Its fixed test set includes attacks that are absent from training, but it remains a legacy benchmark and does not represent modern production traffic."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "cell-012",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"✓ Lab 1 completed\n"
]
}
],
"source": [
"assert len(feature_cols) == 41\n",
"assert set(CATEGORICAL_FEATURES).issubset(feature_cols)\n",
"assert set(train_df[\"binary_label\"].unique()) == {0, 1}\n",
"assert set(test_df[\"attack_group\"].unique()).issubset({\"Normal\", \"DoS\", \"Probe\", \"R2L\", \"U2R\"})\n",
"assert len(unseen_test_attacks) > 0\n",
"assert set(feature_cols).isdisjoint(TARGET_COLUMNS)\n",
"assert quality_summary.shape[0] == 41\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why keep the fixed split?** It preserves the benchmark design and exposes attack labels that training never saw.\n",
"2. **Why exclude `difficulty`?** It was derived from previous classifiers and is not a network measurement available to a real detector.\n",
"3. **Why inspect group counts?** Binary metrics can hide poor detection of rare R2L or U2R records."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/02_Flow_Features_Preprocessing_Solution.ipynb
+224
View File
@@ -0,0 +1,224 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
"\n",
"\n",
"**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- interpret basic, content, traffic, and host-based flow features;\n",
"- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
"- compare normal and attack distributions;\n",
"- build separate preprocessing plans for distance models and tree models;\n",
"- fit encoders and scalers using training data only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
"test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
"feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
"numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
"\n",
"feature_families = {\n",
" \"basic\": [\"duration\", \"protocol_type\", \"service\", \"flag\", \"src_bytes\", \"dst_bytes\", \"land\", \"wrong_fragment\", \"urgent\"],\n",
" \"content\": numeric_features[6:19],\n",
" \"traffic\": numeric_features[19:28],\n",
" \"host_based\": numeric_features[28:],\n",
"}\n",
"{family: len(names) for family, names in feature_families.items()}"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect byte-count skew and apply `log1p`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"exploratory = train_df[[\"src_bytes\", \"dst_bytes\", \"binary_label\"]].copy()\n",
"exploratory[\"log_src_bytes\"] = np.log1p(exploratory[\"src_bytes\"])\n",
"exploratory[\"log_dst_bytes\"] = np.log1p(exploratory[\"dst_bytes\"])\n",
"\n",
"byte_summary = exploratory.groupby(\"binary_label\")[[\n",
" \"src_bytes\", \"dst_bytes\", \"log_src_bytes\", \"log_dst_bytes\"\n",
"]].median()\n",
"display(byte_summary)\n",
"\n",
"sample = exploratory.sample(min(5000, len(exploratory)), random_state=42)\n",
"colors = sample[\"binary_label\"].map({0: \"#4C78A8\", 1: \"#E45756\"})\n",
"ax = sample.plot.scatter(\n",
" x=\"log_src_bytes\", y=\"log_dst_bytes\", c=colors, alpha=0.25, figsize=(8, 6)\n",
")\n",
"ax.set_title(\"NSL-KDD flows after log1p byte transformation\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build preprocessing plans for two model families"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"supervised_preprocessor = ColumnTransformer([\n",
" (\"categorical\", OneHotEncoder(handle_unknown=\"ignore\"), CATEGORICAL_FEATURES),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"\n",
"distance_features = [\n",
" \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
" \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
"]\n",
"distance_preprocessor = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
" (\"scaler\", StandardScaler()),\n",
"])\n",
"\n",
"supervised_preprocessor.fit(train_df[feature_cols])\n",
"distance_preprocessor.fit(train_df[distance_features])\n",
"supervised_shape = supervised_preprocessor.transform(train_df[feature_cols].head(100)).shape\n",
"distance_shape = distance_preprocessor.transform(train_df[distance_features].head(100)).shape\n",
"print(\"Supervised transformed shape:\", supervised_shape)\n",
"print(\"Distance transformed shape:\", distance_shape)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Why two plans?\n",
"\n",
"Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert len(feature_cols) == 41\n",
"assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
"assert supervised_shape[0] == 100\n",
"assert supervised_shape[1] > len(feature_cols)\n",
"assert distance_shape == (100, len(distance_features))\n",
"assert all(feature in numeric_features for feature in distance_features)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why `log1p`?** It compresses large byte values and remains defined at zero.\n",
"2. **Why scale distance features?** Otherwise byte counts can dominate rate features in Euclidean distance.\n",
"3. **Why fit on training only?** Test categories and distribution statistics must not influence the learned preprocessing state."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/03_kNN_Anomaly_Scoring_Solution.ipynb
+430
View File
File diff suppressed because one or more lines are too long
Lab-6/Lab-6-Solution/04_OneClass_IsolationForest_Solution.ipynb
+268
View File
@@ -0,0 +1,268 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 4: One-Class SVM and Isolation Forest\n",
"\n",
"\n",
"**Topics:** normal-only learning, anomaly scores, validation thresholds, PR-AUC, runtime\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- fit two normal-only novelty detectors;\n",
"- standardize features for One-Class SVM;\n",
"- derive a common anomaly-score convention;\n",
"- compare detectors with the same normal-validation alert budget;\n",
"- explain performance and scalability trade-offs."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.svm import OneClassSVM\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
"test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
"anomaly_features = [\n",
" \"duration\", \"src_bytes\", \"dst_bytes\", \"wrong_fragment\", \"hot\",\n",
" \"num_failed_logins\", \"logged_in\", \"count\", \"srv_count\",\n",
" \"serror_rate\", \"rerror_rate\", \"same_srv_rate\", \"diff_srv_rate\",\n",
" \"dst_host_count\", \"dst_host_srv_count\", \"dst_host_same_srv_rate\",\n",
" \"dst_host_diff_srv_rate\", \"dst_host_serror_rate\", \"dst_host_rerror_rate\",\n",
"]\n",
"normal = train_df.loc[train_df[\"binary_label\"] == 0, anomaly_features]\n",
"normal_fit, normal_valid = train_test_split(normal, test_size=0.20, random_state=42)\n",
"normal_fit = normal_fit.sample(min(5000, len(normal_fit)), random_state=42)\n",
"X_test = test_df[anomaly_features]\n",
"y_test = test_df[\"binary_label\"].astype(int)\n",
"print(\"Normal fit/validation/test:\", len(normal_fit), len(normal_valid), len(X_test))"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit the two detectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"detectors = {\n",
" \"One-Class SVM\": Pipeline([\n",
" (\"scaler\", StandardScaler()),\n",
" (\"model\", OneClassSVM(kernel=\"rbf\", gamma=\"scale\", nu=0.02)),\n",
" ]),\n",
" \"Isolation Forest\": Pipeline([\n",
" (\"scaler\", StandardScaler()),\n",
" (\"model\", IsolationForest(\n",
" n_estimators=200, max_samples=\"auto\", contamination=\"auto\",\n",
" random_state=42, n_jobs=-1,\n",
" )),\n",
" ]),\n",
"}\n",
"\n",
"for detector in detectors.values():\n",
" detector.fit(normal_fit)\n",
"print(\"Fitted detectors:\", list(detectors))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Apply a common validation-threshold policy"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"comparison_rows = []\n",
"detector_outputs = {}\n",
"\n",
"for name, detector in detectors.items():\n",
" validation_scores = -detector.decision_function(normal_valid)\n",
" test_scores = -detector.decision_function(X_test)\n",
" threshold = float(np.quantile(validation_scores, 0.99))\n",
" predictions = (test_scores > threshold).astype(int)\n",
" metrics = security_metrics(y_test, predictions)\n",
" metrics[\"pr_auc\"] = average_precision_score(y_test, test_scores)\n",
" comparison_rows.append({\"detector\": name, \"threshold\": threshold, **metrics})\n",
" detector_outputs[name] = {\n",
" \"validation_scores\": validation_scores,\n",
" \"test_scores\": test_scores,\n",
" \"predictions\": predictions,\n",
" }\n",
"\n",
"detector_comparison = pd.DataFrame(comparison_rows).set_index(\"detector\")\n",
"display(detector_comparison.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Compare attack-group recall"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"group_rows = []\n",
"for name, output in detector_outputs.items():\n",
" for group in [\"DoS\", \"Probe\", \"R2L\", \"U2R\"]:\n",
" mask = test_df[\"attack_group\"].eq(group).to_numpy()\n",
" group_rows.append({\n",
" \"detector\": name,\n",
" \"attack_group\": group,\n",
" \"support\": int(mask.sum()),\n",
" \"recall\": float(output[\"predictions\"][mask].mean()) if mask.any() else np.nan,\n",
" })\n",
"group_recall = pd.DataFrame(group_rows).pivot(\n",
" index=\"attack_group\", columns=\"detector\", values=\"recall\"\n",
")\n",
"display(group_recall.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(detectors) == {\"One-Class SVM\", \"Isolation Forest\"}\n",
"assert len(normal_fit) <= 5000\n",
"assert all(len(output[\"test_scores\"]) == len(test_df) for output in detector_outputs.values())\n",
"assert detector_comparison.shape[0] == 2\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert detector_comparison[\"false_positive_rate\"].between(0, 1).all()\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why cap One-Class SVM training?** The RBF implementation scales poorly as the number of training samples grows.\n",
"2. **Why negate `decision_function`?** Both models then follow one convention: larger values mean more anomalous.\n",
"3. **Why can rare-group recall differ?** Different attacks occupy different regions of feature space, and some resemble normal traffic in the selected features."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/05_Supervised_Intrusion_Tree_Solution.ipynb
+248
View File
@@ -0,0 +1,248 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 5: A Supervised Intrusion Decision Tree\n",
"\n",
"\n",
"**Topics:** known-attack classification, fixed test set, tree rules, group recall\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- train a binary classifier from labeled normal and attack records;\n",
"- apply categorical encoding inside a pipeline;\n",
"- evaluate the fixed test set without tuning on it;\n",
"- inspect Decision Tree rules and feature importance;\n",
"- compare aggregate attack recall with group-level recall."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"from sklearn.tree import DecisionTreeClassifier, export_text\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
"test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
"feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
"numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
"X_train, y_train = train_df[feature_cols], train_df[\"binary_label\"].astype(int)\n",
"X_test, y_test = test_df[feature_cols], test_df[\"binary_label\"].astype(int)\n",
"\n",
"preprocessor = ColumnTransformer([\n",
" (\"categorical\", OneHotEncoder(handle_unknown=\"ignore\"), CATEGORICAL_FEATURES),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"tree_pipeline = Pipeline([\n",
" (\"preprocessor\", preprocessor),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=8, min_samples_leaf=5, class_weight=\"balanced\", random_state=42\n",
" )),\n",
"])\n",
"tree_pipeline.fit(X_train, y_train)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Evaluate binary and attack-group performance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"tree_predictions = tree_pipeline.predict(X_test)\n",
"tree_probabilities = tree_pipeline.predict_proba(X_test)[:, 1]\n",
"tree_metrics = security_metrics(y_test, tree_predictions)\n",
"tree_metrics[\"pr_auc\"] = average_precision_score(y_test, tree_probabilities)\n",
"display(pd.Series(tree_metrics, name=\"Decision Tree\"))\n",
"\n",
"group_rows = []\n",
"for group in [\"DoS\", \"Probe\", \"R2L\", \"U2R\"]:\n",
" mask = test_df[\"attack_group\"].eq(group).to_numpy()\n",
" group_rows.append({\n",
" \"attack_group\": group,\n",
" \"support\": int(mask.sum()),\n",
" \"recall\": float(tree_predictions[mask].mean()) if mask.any() else np.nan,\n",
" })\n",
"group_recall = pd.DataFrame(group_rows).set_index(\"attack_group\")\n",
"display(group_recall.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Interpret the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"fitted_preprocessor = tree_pipeline.named_steps[\"preprocessor\"]\n",
"fitted_tree = tree_pipeline.named_steps[\"tree\"]\n",
"transformed_names = fitted_preprocessor.get_feature_names_out()\n",
"importance = pd.Series(\n",
" fitted_tree.feature_importances_, index=transformed_names, name=\"importance\"\n",
").sort_values(ascending=False)\n",
"top_importance = importance.head(12)\n",
"display(top_importance.to_frame())\n",
"\n",
"rule_text = export_text(fitted_tree, feature_names=list(transformed_names), max_depth=3)\n",
"print(rule_text)\n",
"\n",
"ax = top_importance.sort_values().plot(kind=\"barh\", figsize=(9, 6), color=\"#4C78A8\")\n",
"ax.set_title(\"Top Decision Tree feature importances\")\n",
"ax.set_xlabel(\"Mean decrease in impurity\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"High performance on NSL-KDD does not establish production readiness. Tree rules can reflect benchmark artifacts, and attack-group recall can expose failures hidden by the binary score."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 8\n",
"assert len(tree_predictions) == len(test_df)\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **What does the tree learn that novelty models do not?** It directly uses labeled attack patterns and can learn boundaries around known attacks.\n",
"2. **Why inspect group recall?** A high binary score may be dominated by common DoS attacks while rare R2L/U2R attacks remain undetected.\n",
"3. **Are tree rules signatures?** No. They describe one model fitted to one historical benchmark and require external validation."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/06_Capstone_Hybrid_IDS_Solution.ipynb
+395
View File
@@ -0,0 +1,395 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 6: Capstone — Hybrid Network Intrusion Triage\n",
"\n",
"\n",
"**Topics:** classifier, anomaly score, hybrid policy, alert budget, SOC report\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Compare three alert policies on the fixed NSL-KDD test set:\n",
"\n",
"1. a supervised Decision Tree for known attack patterns;\n",
"2. a normal-only Isolation Forest for anomaly hunting;\n",
"3. a hybrid OR policy that alerts when either detector fires.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/hybrid_ids_metrics.csv`\n",
"- `output/hybrid_ids_group_recall.csv`\n",
"- `output/hybrid_ids_top_alerts.csv`\n",
"- `output/hybrid_ids_confusion_matrix.png`\n",
"- `output/hybrid_ids_score_distribution.png`\n",
"- `output/hybrid_ids_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"train_df = pd.read_csv(TRAIN_PATH, low_memory=False)\n",
"test_df = pd.read_csv(TEST_PATH, low_memory=False)\n",
"feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]\n",
"numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]\n",
"X_train, y_train = train_df[feature_cols], train_df[\"binary_label\"].astype(int)\n",
"X_test, y_test = test_df[feature_cols], test_df[\"binary_label\"].astype(int)\n",
"\n",
"supervised_preprocessor = ColumnTransformer([\n",
" (\"categorical\", OneHotEncoder(handle_unknown=\"ignore\"), CATEGORICAL_FEATURES),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"tree_model = Pipeline([\n",
" (\"preprocessor\", supervised_preprocessor),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=8, min_samples_leaf=5, class_weight=\"balanced\", random_state=42\n",
" )),\n",
"])\n",
"tree_model.fit(X_train, y_train)\n",
"tree_predictions = tree_model.predict(X_test)\n",
"tree_probabilities = tree_model.predict_proba(X_test)[:, 1]"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit a normal-only Isolation Forest and set its alert budget"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"anomaly_features = [\n",
" \"duration\", \"src_bytes\", \"dst_bytes\", \"wrong_fragment\", \"hot\",\n",
" \"num_failed_logins\", \"logged_in\", \"count\", \"srv_count\",\n",
" \"serror_rate\", \"rerror_rate\", \"same_srv_rate\", \"diff_srv_rate\",\n",
" \"dst_host_count\", \"dst_host_srv_count\", \"dst_host_same_srv_rate\",\n",
" \"dst_host_diff_srv_rate\", \"dst_host_serror_rate\", \"dst_host_rerror_rate\",\n",
"]\n",
"normal = train_df.loc[train_df[\"binary_label\"] == 0, anomaly_features]\n",
"normal_fit, normal_valid = train_test_split(normal, test_size=0.20, random_state=42)\n",
"\n",
"anomaly_model = Pipeline([\n",
" (\"scaler\", StandardScaler()),\n",
" (\"model\", IsolationForest(\n",
" n_estimators=250, contamination=\"auto\", random_state=42, n_jobs=-1\n",
" )),\n",
"])\n",
"anomaly_model.fit(normal_fit)\n",
"validation_scores = -anomaly_model.decision_function(normal_valid)\n",
"anomaly_scores = -anomaly_model.decision_function(test_df[anomaly_features])\n",
"anomaly_threshold = float(np.quantile(validation_scores, 0.99))\n",
"anomaly_predictions = (anomaly_scores > anomaly_threshold).astype(int)\n",
"print(\"Anomaly threshold:\", anomaly_threshold)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Compare classifier, anomaly, and hybrid policies"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"policy_predictions = {\n",
" \"Classifier\": tree_predictions,\n",
" \"Anomaly\": anomaly_predictions,\n",
" \"Hybrid_OR\": np.logical_or(tree_predictions == 1, anomaly_predictions == 1).astype(int),\n",
"}\n",
"policy_scores = {\n",
" \"Classifier\": tree_probabilities,\n",
" \"Anomaly\": anomaly_scores,\n",
" \"Hybrid_OR\": np.maximum(\n",
" tree_probabilities,\n",
" np.clip((anomaly_scores - anomaly_threshold) + 0.5, 0, 1),\n",
" ),\n",
"}\n",
"\n",
"metric_rows = []\n",
"for name, predictions in policy_predictions.items():\n",
" metrics = security_metrics(y_test, predictions)\n",
" metrics[\"pr_auc\"] = average_precision_score(y_test, policy_scores[name])\n",
" metric_rows.append({\"policy\": name, **metrics})\n",
"metrics_table = pd.DataFrame(metric_rows).set_index(\"policy\")\n",
"display(metrics_table.round(4))\n",
"\n",
"acceptable = metrics_table[metrics_table[\"false_positive_rate\"] <= 0.10]\n",
"selection_pool = acceptable if not acceptable.empty else metrics_table\n",
"selected_policy = selection_pool.sort_values(\n",
" [\"attack_recall\", \"attack_f1\", \"false_positive_rate\"],\n",
" ascending=[False, False, True],\n",
").index[0]\n",
"selected_predictions = policy_predictions[selected_policy]\n",
"print(\"Selected policy:\", selected_policy)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Audit attack groups and create the alert evidence table"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"group_rows = []\n",
"for policy, predictions in policy_predictions.items():\n",
" for group in [\"DoS\", \"Probe\", \"R2L\", \"U2R\"]:\n",
" mask = test_df[\"attack_group\"].eq(group).to_numpy()\n",
" group_rows.append({\n",
" \"policy\": policy,\n",
" \"attack_group\": group,\n",
" \"support\": int(mask.sum()),\n",
" \"recall\": float(predictions[mask].mean()) if mask.any() else np.nan,\n",
" })\n",
"group_recall = pd.DataFrame(group_rows)\n",
"display(group_recall.pivot(index=\"attack_group\", columns=\"policy\", values=\"recall\").round(4))\n",
"\n",
"alert_table = test_df[[\n",
" \"protocol_type\", \"service\", \"flag\", \"src_bytes\", \"dst_bytes\",\n",
" \"count\", \"srv_count\", \"attack_label\", \"attack_group\", \"binary_label\",\n",
"]].copy()\n",
"alert_table[\"tree_probability\"] = tree_probabilities\n",
"alert_table[\"anomaly_score\"] = anomaly_scores\n",
"alert_table[\"tree_alert\"] = tree_predictions\n",
"alert_table[\"anomaly_alert\"] = anomaly_predictions\n",
"alert_table[\"selected_alert\"] = selected_predictions\n",
"alert_table[\"alert_reason\"] = np.select(\n",
" [\n",
" (tree_predictions == 1) & (anomaly_predictions == 1),\n",
" tree_predictions == 1,\n",
" anomaly_predictions == 1,\n",
" ],\n",
" [\"classifier_and_anomaly\", \"classifier_only\", \"anomaly_only\"],\n",
" default=\"no_alert\",\n",
")\n",
"top_alerts = alert_table.loc[alert_table[\"selected_alert\"] == 1].sort_values(\n",
" [\"anomaly_score\", \"tree_probability\"], ascending=False\n",
").head(200)\n",
"display(top_alerts.head(10))"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export evidence and write the SOC recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"fig, ax = plt.subplots(figsize=(6, 5))\n",
"ConfusionMatrixDisplay.from_predictions(\n",
" y_test, selected_predictions, labels=[0, 1],\n",
" display_labels=[\"normal\", \"attack\"], cmap=\"Blues\", colorbar=False, ax=ax,\n",
")\n",
"ax.set_title(f\"{selected_policy} policy — fixed test set\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"hybrid_ids_confusion_matrix.png\", dpi=160)\n",
"plt.show()\n",
"\n",
"clipped_scores = np.clip(anomaly_scores, None, np.quantile(anomaly_scores, 0.995))\n",
"fig, ax = plt.subplots(figsize=(9, 5))\n",
"ax.hist(clipped_scores[y_test.to_numpy() == 0], bins=60, alpha=0.6, density=True, label=\"Normal\")\n",
"ax.hist(clipped_scores[y_test.to_numpy() == 1], bins=60, alpha=0.5, density=True, label=\"Attack\")\n",
"ax.axvline(anomaly_threshold, color=\"black\", linestyle=\"--\", label=\"Threshold\")\n",
"ax.set_title(\"Isolation Forest anomaly-score distribution\")\n",
"ax.set_xlabel(\"Anomaly score\")\n",
"ax.set_ylabel(\"Density\")\n",
"ax.legend()\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"hybrid_ids_score_distribution.png\", dpi=160)\n",
"plt.show()\n",
"\n",
"selected_metrics = metrics_table.loc[selected_policy]\n",
"analyst_summary = \"\\n\".join([\n",
" \"# Hybrid IDS Capstone — SOC Analyst Summary\", \"\",\n",
" \"## Recommendation\", \"\",\n",
" f\"Selected policy: **{selected_policy}**.\", \"\",\n",
" f\"- Attack recall: {selected_metrics['attack_recall']:.3f}\",\n",
" f\"- Attack precision: {selected_metrics['attack_precision']:.3f}\",\n",
" f\"- Attack F1: {selected_metrics['attack_f1']:.3f}\",\n",
" f\"- False-positive rate: {selected_metrics['false_positive_rate']:.3f}\",\n",
" f\"- Alerts per 10,000 flows: {selected_metrics['alerts_per_10000']:.1f}\",\n",
" f\"- False negatives: {int(selected_metrics['false_negatives'])}\", \"\",\n",
" \"## Operating model\", \"\",\n",
" \"The classifier handles patterns represented by labeled training attacks. The anomaly detector adds coverage for flows that deviate from the learned normal baseline. The hybrid policy can improve recall, but every added anomaly alert consumes analyst capacity.\", \"\",\n",
" \"## Limitations\", \"\",\n",
" \"NSL-KDD is a legacy benchmark with synthetic historical traffic and unrealistic prevalence. The results do not demonstrate readiness for encrypted enterprise traffic, current botnets, concept drift, or adversarial evasion. Production use would require current telemetry, time-aware validation, calibration, monitoring, and layered signature or sandbox evidence.\", \"\",\n",
"])\n",
"\n",
"metrics_table.to_csv(OUTPUT_DIR / \"hybrid_ids_metrics.csv\")\n",
"group_recall.to_csv(OUTPUT_DIR / \"hybrid_ids_group_recall.csv\", index=False)\n",
"top_alerts.to_csv(OUTPUT_DIR / \"hybrid_ids_top_alerts.csv\", index=False)\n",
"(OUTPUT_DIR / \"hybrid_ids_analyst_summary.md\").write_text(analyst_summary, encoding=\"utf-8\")\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"hybrid_ids_metrics.csv\", \"hybrid_ids_group_recall.csv\",\n",
" \"hybrid_ids_top_alerts.csv\", \"hybrid_ids_confusion_matrix.png\",\n",
" \"hybrid_ids_score_distribution.png\", \"hybrid_ids_analyst_summary.md\",\n",
"}\n",
"assert set(policy_predictions) == {\"Classifier\", \"Anomaly\", \"Hybrid_OR\"}\n",
"assert selected_policy in policy_predictions\n",
"assert len(alert_table) == len(test_df)\n",
"assert set(group_recall[\"attack_group\"]) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"NSL-KDD\" in analyst_summary\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Suggested review checklist\n",
"\n",
"- The anomaly model learned from normal training records only.\n",
"- The anomaly threshold came from normal validation scores.\n",
"- The fixed test set influenced evaluation, not training or threshold selection.\n",
"- The recommendation balances attack recall with false-positive rate and alert volume.\n",
"- Per-group recall accompanies the aggregate binary score.\n",
"- The report separates known-pattern classification from anomaly hunting.\n",
"- The report states that NSL-KDD performance is not a production guarantee."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/07_Botnet_Flow_Detection_CTU13_Solution.ipynb
+477
View File
@@ -0,0 +1,477 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n",
"\n",
" \n",
"**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n",
"\n",
"The workflow deliberately separates the three CTU-13 label families:\n",
"\n",
"- **Botnet:** verified traffic produced by an infected host;\n",
"- **Normal:** verified traffic produced by known normal hosts;\n",
"- **Background:** traffic whose status is not known with confidence.\n",
"\n",
"Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/ctu13_botnet_metrics.csv`\n",
"- `output/ctu13_source_triage.csv`\n",
"- `output/ctu13_top_flows.csv`\n",
"- `output/ctu13_confusion_matrix.png`\n",
"- `output/ctu13_score_distribution.png`\n",
"- `output/ctu13_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n",
" \"python data/prepare_ctu13.py <path-to-scenario.binetflow>\"\n",
" )\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n",
" confusion_matrix, f1_score, precision_score, recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"\n",
"RANDOM_STATE = 42"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Audit the prepared scenario\n",
"\n",
"Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"flows = pd.read_csv(DATA_PATH, low_memory=False, parse_dates=[\"start_time\"])\n",
"expected_families = {\"Botnet\", \"Normal\", \"Background\"}\n",
"print(\"Shape:\", flows.shape)\n",
"display(flows.head(3))\n",
"display(flows[\"label_family\"].value_counts().rename(\"flows\").to_frame())\n",
"print(\"Raw label examples:\")\n",
"for value in flows[\"raw_label\"].dropna().drop_duplicates().head(8):\n",
" print(\" -\", value)"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"### Suggested answers\n",
"\n",
"1. Background means the traffic could not be confidently assigned to the verified Botnet or Normal hosts. It is uncertain, not confirmed benign.\n",
"2. Addresses can identify scenario-specific infected hosts, while label fields directly encode the target. Either shortcut would inflate performance without learning behavior.\n",
"3. The preparation utility reports retained counts. All three families are expected for the recommended scenario, although only Botnet and Normal enter supervised metrics."
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Define behavioral features and create three data splits\n",
"\n",
"Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n",
"\n",
"The validation set selects an operating threshold. The test set remains untouched until the final evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"categorical_features = [\"protocol\", \"direction\", \"state\"]\n",
"numeric_features = [\n",
" \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n",
" \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n",
" \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n",
" \"is_well_known_dst_port\",\n",
"]\n",
"feature_columns = categorical_features + numeric_features\n",
"\n",
"\n",
"labeled = flows.loc[flows[\"label_family\"].isin([\"Botnet\", \"Normal\"])].copy()\n",
"background = flows.loc[flows[\"label_family\"].eq(\"Background\")].copy()\n",
"X = labeled[feature_columns]\n",
"y = labeled[\"binary_label\"].astype(int)\n",
"\n",
"X_train_valid, X_test, y_train_valid, y_test = train_test_split(\n",
" X, y, test_size=0.20, stratify=y, random_state=RANDOM_STATE\n",
")\n",
"X_train, X_valid, y_train, y_valid = train_test_split(\n",
" X_train_valid, y_train_valid, test_size=0.25,\n",
" stratify=y_train_valid, random_state=RANDOM_STATE,\n",
")\n",
"\n",
"print(\"Train / validation / test:\", len(X_train), len(X_valid), len(X_test))\n",
"print(\"Background records reserved for triage:\", len(background))"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Train a bounded Random Forest\n",
"\n",
"The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"preprocessor = ColumnTransformer([\n",
" (\"categorical\", Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
" (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
" ]), categorical_features),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"\n",
"detector = Pipeline([\n",
" (\"preprocessor\", preprocessor),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=200,\n",
" max_depth=12,\n",
" min_samples_leaf=3,\n",
" class_weight=\"balanced_subsample\",\n",
" random_state=RANDOM_STATE,\n",
" n_jobs=-1,\n",
" )),\n",
"])\n",
"\n",
"\n",
"detector.fit(X_train, y_train)\n",
"validation_scores = detector.predict_proba(X_valid)[:, 1]"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Select a validation threshold under an alert-quality constraint\n",
"\n",
"Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"threshold_rows = []\n",
"for threshold in np.linspace(0.05, 0.95, 91):\n",
" predictions = (validation_scores >= threshold).astype(int)\n",
" tn, fp, fn, tp = confusion_matrix(y_valid, predictions, labels=[0, 1]).ravel()\n",
" threshold_rows.append({\n",
" \"threshold\": threshold,\n",
" \"botnet_recall\": recall_score(y_valid, predictions, zero_division=0),\n",
" \"botnet_precision\": precision_score(y_valid, predictions, zero_division=0),\n",
" \"botnet_f1\": f1_score(y_valid, predictions, zero_division=0),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts\": int(predictions.sum()),\n",
" })\n",
"\n",
"threshold_table = pd.DataFrame(threshold_rows)\n",
"acceptable = threshold_table.loc[threshold_table[\"false_positive_rate\"] <= 0.05]\n",
"selection_pool = acceptable if not acceptable.empty else threshold_table\n",
"selected_row = selection_pool.sort_values(\n",
" [\"botnet_recall\", \"botnet_f1\", \"false_positive_rate\"],\n",
" ascending=[False, False, True],\n",
").iloc[0]\n",
"operating_threshold = float(selected_row[\"threshold\"])\n",
"display(selected_row.to_frame(\"selected_value\"))"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## 5. Evaluate once on the held-out test set\n",
"\n",
"Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"test_scores = detector.predict_proba(X_test)[:, 1]\n",
"test_predictions = (test_scores >= operating_threshold).astype(int)\n",
"tn, fp, fn, tp = confusion_matrix(y_test, test_predictions, labels=[0, 1]).ravel()\n",
"metrics_table = pd.DataFrame([{\n",
" \"threshold\": operating_threshold,\n",
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
" \"botnet_precision\": precision_score(y_test, test_predictions, zero_division=0),\n",
" \"botnet_recall\": recall_score(y_test, test_predictions, zero_division=0),\n",
" \"botnet_f1\": f1_score(y_test, test_predictions, zero_division=0),\n",
" \"pr_auc\": average_precision_score(y_test, test_scores),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"alerts_per_10000\": 10000 * int(test_predictions.sum()) / len(y_test),\n",
"}])\n",
"display(metrics_table.round(4))\n",
"\n",
"fig, ax = plt.subplots(figsize=(6, 5))\n",
"ConfusionMatrixDisplay.from_predictions(\n",
" y_test, test_predictions, labels=[0, 1],\n",
" display_labels=[\"Normal\", \"Botnet\"], cmap=\"Blues\", colorbar=False, ax=ax,\n",
")\n",
"ax.set_title(\"CTU-13 Botnet detector — held-out labeled flows\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"ctu13_confusion_matrix.png\", dpi=160)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## 6. Score all flows and create source-level triage\n",
"\n",
"Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n",
"\n",
"A high Background score is a lead for investigation, not proof that the flow is malicious."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-016",
"metadata": {},
"outputs": [],
"source": [
"all_scores = detector.predict_proba(flows[feature_columns])[:, 1]\n",
"scored_flows = flows.copy()\n",
"scored_flows[\"botnet_score\"] = all_scores\n",
"scored_flows[\"botnet_alert\"] = (all_scores >= operating_threshold).astype(int)\n",
"\n",
"source_triage = (\n",
" scored_flows.groupby(\"src_addr\", dropna=False)\n",
" .agg(\n",
" flow_count=(\"src_addr\", \"size\"),\n",
" mean_botnet_score=(\"botnet_score\", \"mean\"),\n",
" max_botnet_score=(\"botnet_score\", \"max\"),\n",
" botnet_alerts=(\"botnet_alert\", \"sum\"),\n",
" observed_botnet_labels=(\"binary_label\", lambda values: int((values == 1).sum())),\n",
" uncertain_background_flows=(\"binary_label\", lambda values: int((values == -1).sum())),\n",
" )\n",
" .reset_index()\n",
")\n",
"source_triage[\"alert_rate\"] = (\n",
" source_triage[\"botnet_alerts\"] / source_triage[\"flow_count\"]\n",
")\n",
"source_triage = source_triage.sort_values(\n",
" [\"botnet_alerts\", \"max_botnet_score\", \"flow_count\"], ascending=False\n",
")\n",
"\n",
"evidence_columns = [\n",
" \"start_time\", \"src_addr\", \"dst_addr\", \"protocol\", \"src_port\", \"dst_port\",\n",
" \"duration\", \"total_packets\", \"total_bytes\", \"label_family\", \"raw_label\",\n",
" \"botnet_score\", \"botnet_alert\",\n",
"]\n",
"top_flows = scored_flows.loc[scored_flows[\"botnet_alert\"].eq(1), evidence_columns].sort_values(\n",
" \"botnet_score\", ascending=False\n",
").head(500)\n",
"display(source_triage.head(10))\n",
"display(top_flows.head(10))\n",
"\n",
"fig, ax = plt.subplots(figsize=(9, 5))\n",
"for family, color in [(\"Normal\", \"tab:green\"), (\"Botnet\", \"tab:red\"), (\"Background\", \"tab:gray\")]:\n",
" values = scored_flows.loc[scored_flows[\"label_family\"].eq(family), \"botnet_score\"]\n",
" if not values.empty:\n",
" ax.hist(values, bins=50, alpha=0.45, density=True, label=family, color=color)\n",
"ax.axvline(operating_threshold, color=\"black\", linestyle=\"--\", label=\"Operating threshold\")\n",
"ax.set_title(\"CTU-13 botnet-score distribution\")\n",
"ax.set_xlabel(\"Predicted Botnet probability\")\n",
"ax.set_ylabel(\"Density\")\n",
"ax.legend()\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"ctu13_score_distribution.png\", dpi=160)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-017",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n",
"\n",
"The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-018",
"metadata": {},
"outputs": [],
"source": [
"metrics_table.to_csv(OUTPUT_DIR / \"ctu13_botnet_metrics.csv\", index=False)\n",
"source_triage.head(500).to_csv(OUTPUT_DIR / \"ctu13_source_triage.csv\", index=False)\n",
"top_flows.to_csv(OUTPUT_DIR / \"ctu13_top_flows.csv\", index=False)\n",
"\n",
"result = metrics_table.iloc[0]\n",
"background_alerts = int(\n",
" scored_flows.loc[scored_flows[\"label_family\"].eq(\"Background\"), \"botnet_alert\"].sum()\n",
")\n",
"analyst_summary = \"\\n\".join([\n",
" \"# CTU-13 Botnet Flow Detection — Analyst Summary\", \"\",\n",
" \"## Labeled-flow evaluation\", \"\",\n",
" f\"- Operating threshold selected on validation data: {operating_threshold:.2f}\",\n",
" f\"- Botnet precision: {result['botnet_precision']:.3f}\",\n",
" f\"- Botnet recall: {result['botnet_recall']:.3f}\",\n",
" f\"- Botnet F1: {result['botnet_f1']:.3f}\",\n",
" f\"- PR-AUC: {result['pr_auc']:.3f}\",\n",
" f\"- False-positive rate: {result['false_positive_rate']:.3f}\", \"\",\n",
" \"## Triage\", \"\",\n",
" f\"- Background flows above threshold: {background_alerts}\",\n",
" f\"- Highest-priority source: {source_triage.iloc[0]['src_addr']}\",\n",
" \"- Background is uncertain traffic; an alert is an investigation lead, not verified ground truth.\", \"\",\n",
" \"## Limitations\", \"\",\n",
" \"This experiment uses a bounded sample from one CTU-13 scenario and a random stratified split.\",\n",
" \"Its result may be optimistic for the same botnet family and network environment.\",\n",
" \"A stronger follow-up trains on one or more scenarios and tests on a different scenario.\",\n",
" \"IP addresses were excluded from model inputs and used only for analyst aggregation.\",\n",
"])\n",
"(OUTPUT_DIR / \"ctu13_analyst_summary.md\").write_text(analyst_summary, encoding=\"utf-8\")\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-019",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-020",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n",
" \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n",
" \"ctu13_analyst_summary.md\",\n",
"}\n",
"assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n",
"assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n",
"assert set(y_test.unique()) == {0, 1}\n",
"assert 0.0 <= operating_threshold <= 1.0\n",
"assert len(test_scores) == len(y_test)\n",
"assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 7 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-021",
"metadata": {},
"source": [
"## Suggested review checklist\n",
"\n",
"- Botnet and Normal provide supervised ground truth; Background remains uncertain.\n",
"- IP addresses and target-derived fields are not model inputs.\n",
"- Training, validation, and test records are separated before model fitting.\n",
"- The validation set selects the operating threshold; test labels are used once.\n",
"- Test reporting includes recall, precision, F1, PR-AUC, false-positive rate, and alert volume.\n",
"- Source aggregation supports investigation without letting the model memorize infected hosts.\n",
"- The report identifies the optimistic same-scenario split and recommends cross-scenario evaluation."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Solution/data/README.md
+174
View File
@@ -0,0 +1,174 @@
# Dataset setup — NSL-KDD
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
## Download
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
```text
KDDTrain+_20Percent.txt
KDDTest+.txt
```
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
Place the files as follows:
```text
Practical-6/
└── data/
├── prepare_dataset.py
└── raw/
├── KDDTrain+_20Percent.txt
└── KDDTest+.txt
```
## Prepare the classroom tables
From `Practical-6`, run:
```bash
python data/prepare_dataset.py
```
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
```bash
python data/prepare_dataset.py --full-train
```
The utility:
- applies the official 41 feature names;
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
- validates numeric and categorical columns;
- rejects unknown attack labels so mapping errors remain visible;
- preserves the original fixed train/test split;
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
print("Train:", train.shape)
print("Test:", test.shape)
print(train["binary_label"].value_counts().sort_index())
print(test["attack_group"].value_counts())
```
Each prepared table must have 41 model features plus these four audit/target columns:
```text
attack_label, attack_group, binary_label, difficulty
```
## Citation and redistribution
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
Retain this citation when redistributing prepared classroom data.
## Important limitations
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
- Class prevalence does not represent a production SOC.
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
- The dataset has no dedicated botnet label.
---
# Dataset setup — CTU-13 for Lab 7
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
## Recommended download
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
```text
capture20110816-3.binetflow
```
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
## Placement and preparation
Place the `.binetflow` file below `data/raw/`:
```text
Practical-6/
└── data/
├── prepare_ctu13.py
└── raw/
└── capture20110816-3.binetflow
```
From `Practical-6`, run:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
```
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
```bash
python data/prepare_ctu13.py
```
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
```
It creates:
```text
data/ctu13_botnet_flows.csv
```
The loader accepts both common CTU-13 flow schemas:
- the compact 15-column export containing `Dir`;
- the extended Argus export containing directional counters such as
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
For the extended schema, the classroom `direction` feature is derived as
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
and destination packet counters. When available, the original `DstBytes` value
is used instead of reconstructing it from total and source bytes.
## Label policy
- `Botnet` receives `binary_label = 1`.
- `Normal` receives `binary_label = 0`.
- `Background` receives `binary_label = -1` because its status is uncertain.
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
## Safety and leakage controls
- Use only `.binetflow` flow records; do not download or execute malware samples.
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
- Raw label fields are excluded from model features.
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
## Citation
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
Lab-6/Lab-6-Solution/data/ctu13_botnet_flows.csv
+46128
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Solution/data/feature_manifest.csv
+42
View File
@@ -0,0 +1,42 @@
feature,feature_type,model_feature
duration,numeric,True
protocol_type,categorical,True
service,categorical,True
flag,categorical,True
src_bytes,numeric,True
dst_bytes,numeric,True
land,numeric,True
wrong_fragment,numeric,True
urgent,numeric,True
hot,numeric,True
num_failed_logins,numeric,True
logged_in,numeric,True
num_compromised,numeric,True
root_shell,numeric,True
su_attempted,numeric,True
num_root,numeric,True
num_file_creations,numeric,True
num_shells,numeric,True
num_access_files,numeric,True
num_outbound_cmds,numeric,True
is_host_login,numeric,True
is_guest_login,numeric,True
count,numeric,True
srv_count,numeric,True
serror_rate,numeric,True
srv_serror_rate,numeric,True
rerror_rate,numeric,True
srv_rerror_rate,numeric,True
same_srv_rate,numeric,True
diff_srv_rate,numeric,True
srv_diff_host_rate,numeric,True
dst_host_count,numeric,True
dst_host_srv_count,numeric,True
dst_host_same_srv_rate,numeric,True
dst_host_diff_srv_rate,numeric,True
dst_host_same_src_port_rate,numeric,True
dst_host_srv_diff_host_rate,numeric,True
dst_host_serror_rate,numeric,True
dst_host_srv_serror_rate,numeric,True
dst_host_rerror_rate,numeric,True
dst_host_srv_rerror_rate,numeric,True
1 feature feature_type model_feature
2 duration numeric True
3 protocol_type categorical True
4 service categorical True
5 flag categorical True
6 src_bytes numeric True
7 dst_bytes numeric True
8 land numeric True
9 wrong_fragment numeric True
10 urgent numeric True
11 hot numeric True
12 num_failed_logins numeric True
13 logged_in numeric True
14 num_compromised numeric True
15 root_shell numeric True
16 su_attempted numeric True
17 num_root numeric True
18 num_file_creations numeric True
19 num_shells numeric True
20 num_access_files numeric True
21 num_outbound_cmds numeric True
22 is_host_login numeric True
23 is_guest_login numeric True
24 count numeric True
25 srv_count numeric True
26 serror_rate numeric True
27 srv_serror_rate numeric True
28 rerror_rate numeric True
29 srv_rerror_rate numeric True
30 same_srv_rate numeric True
31 diff_srv_rate numeric True
32 srv_diff_host_rate numeric True
33 dst_host_count numeric True
34 dst_host_srv_count numeric True
35 dst_host_same_srv_rate numeric True
36 dst_host_diff_srv_rate numeric True
37 dst_host_same_src_port_rate numeric True
38 dst_host_srv_diff_host_rate numeric True
39 dst_host_serror_rate numeric True
40 dst_host_srv_serror_rate numeric True
41 dst_host_rerror_rate numeric True
42 dst_host_srv_rerror_rate numeric True
Lab-6/Lab-6-Solution/data/nsl_kdd_test.csv
+22545
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Solution/data/nsl_kdd_train.csv
+125974
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Solution/data/prepare_ctu13.py
+261
View File
@@ -0,0 +1,261 @@
"""Create a bounded, classroom-ready flow table from one CTU-13 .binetflow file."""
from __future__ import annotations
import argparse
from pathlib import Path
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
REQUIRED_RAW_COLUMNS = [
"StartTime", "Dur", "Proto", "SrcAddr", "Sport", "DstAddr", "Dport",
"State", "sTos", "dTos", "TotPkts", "TotBytes", "SrcBytes", "Label",
]
# CTU-13 is distributed in more than one Argus text schema. The compact
# 15-column export contains ``Dir``. The extended export supplied with some
# scenarios omits it but provides per-direction packet and byte counters.
OPTIONAL_RAW_COLUMNS = ["Dir", "SrcPkts", "DstPkts", "DstBytes"]
RENAME_MAP = {
"StartTime": "start_time",
"Dur": "duration",
"Proto": "protocol",
"SrcAddr": "src_addr",
"Sport": "src_port",
"Dir": "direction",
"DstAddr": "dst_addr",
"Dport": "dst_port",
"State": "state",
"sTos": "src_tos",
"dTos": "dst_tos",
"TotPkts": "total_packets",
"TotBytes": "total_bytes",
"SrcBytes": "src_bytes",
"DstBytes": "dst_bytes",
"Label": "raw_label",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"input",
nargs="?",
type=Path,
help="Path to one labeled CTU-13 .binetflow file. If omitted, data/raw is searched.",
)
parser.add_argument(
"--per-class",
type=int,
default=20_000,
help="Maximum sampled rows for each of Botnet, Normal, and Background.",
)
parser.add_argument(
"--chunk-size",
type=int,
default=200_000,
help="Rows read at a time; reduce this value on low-memory computers.",
)
parser.add_argument(
"--output",
type=Path,
default=HERE / "ctu13_botnet_flows.csv",
help="Prepared CSV destination.",
)
return parser.parse_args()
def find_input(requested: Path | None) -> Path:
if requested is not None:
path = requested.expanduser().resolve()
if not path.is_file():
raise FileNotFoundError(f"CTU-13 input file not found: {path}")
return path
matches = sorted(RAW_DIR.rglob("*.binetflow")) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one .binetflow file below {RAW_DIR}, found {len(matches)}. "
"Pass the input path explicitly when more than one file is present."
)
return matches[0]
def label_family(labels: pd.Series) -> pd.Series:
text = labels.fillna("").astype(str)
return pd.Series(
np.select(
[
text.str.contains("Botnet", case=False, regex=False),
text.str.contains("Normal", case=False, regex=False),
text.str.contains("Background", case=False, regex=False),
],
["Botnet", "Normal", "Background"],
default="Other",
),
index=labels.index,
dtype="string",
)
def parse_port(values: pd.Series) -> pd.Series:
text = values.fillna("").astype(str).str.strip().str.lower()
decimal = pd.to_numeric(text.where(~text.str.startswith("0x")), errors="coerce")
hexadecimal = text.where(text.str.startswith("0x")).map(
lambda value: int(value, 16) if isinstance(value, str) and value else np.nan
)
return decimal.fillna(hexadecimal).astype("Float64")
def inspect_schema(path: Path) -> tuple[list[str], str]:
"""Validate a compact or extended CTU-13 header and select needed fields."""
header = pd.read_csv(path, nrows=0).columns.tolist()
missing = sorted(set(REQUIRED_RAW_COLUMNS) - set(header))
if missing:
raise ValueError(
f"Missing required CTU-13 columns: {missing}. Available columns: {header}"
)
selected = REQUIRED_RAW_COLUMNS + [
column for column in OPTIONAL_RAW_COLUMNS if column in header
]
if "Dir" in header:
direction_mode = "source Dir column"
elif {"SrcPkts", "DstPkts"}.issubset(header):
direction_mode = "derived from SrcPkts/DstPkts"
else:
direction_mode = "constant unknown (no Dir or packet-direction counters)"
print(f"Detected {len(header)}-column CTU-13 schema; direction is {direction_mode}.")
return selected, direction_mode
def ensure_direction(frame: pd.DataFrame) -> pd.DataFrame:
"""Return a frame with a categorical Dir field for either CTU-13 schema."""
if "Dir" in frame.columns:
frame["Dir"] = frame["Dir"].fillna("unknown").astype(str).str.strip()
frame.loc[frame["Dir"].eq(""), "Dir"] = "unknown"
return frame
if {"SrcPkts", "DstPkts"}.issubset(frame.columns):
src_packets = pd.to_numeric(frame["SrcPkts"], errors="coerce").fillna(0)
dst_packets = pd.to_numeric(frame["DstPkts"], errors="coerce").fillna(0)
frame["Dir"] = np.select(
[
(src_packets > 0) & (dst_packets > 0),
(src_packets > 0) & (dst_packets <= 0),
(src_packets <= 0) & (dst_packets > 0),
],
["bidirectional", "source_only", "destination_only"],
default="unknown",
)
else:
frame["Dir"] = "unknown"
return frame
def bounded_sample(path: Path, per_class: int, chunk_size: int) -> pd.DataFrame:
if per_class < 1 or chunk_size < 1:
raise ValueError("--per-class and --chunk-size must be positive integers.")
kept = {family: pd.DataFrame() for family in ("Botnet", "Normal", "Background")}
rng = np.random.default_rng(42)
saw_header = False
selected_columns, _ = inspect_schema(path)
for chunk_number, chunk in enumerate(
pd.read_csv(
path,
usecols=selected_columns,
chunksize=chunk_size,
low_memory=False,
),
start=1,
):
saw_header = True
chunk = ensure_direction(chunk)
output_columns = REQUIRED_RAW_COLUMNS + ["Dir"]
if "DstBytes" in chunk.columns:
output_columns.append("DstBytes")
chunk = chunk[output_columns].copy()
chunk["label_family"] = label_family(chunk["Label"])
for family in kept:
part = chunk.loc[chunk["label_family"].eq(family)].copy()
if part.empty:
continue
part["_sample_key"] = rng.random(len(part))
candidates = pd.concat([kept[family], part], ignore_index=True)
kept[family] = candidates.nsmallest(per_class, "_sample_key")
counts = {family: len(frame) for family, frame in kept.items()}
print(f"Processed chunk {chunk_number}: retained {counts}")
if not saw_header:
raise ValueError(f"No rows found in {path}")
sampled = pd.concat(kept.values(), ignore_index=True).drop(columns="_sample_key")
if sampled.empty:
raise ValueError("No Botnet, Normal, or Background labels were found.")
return sampled
def engineer_features(sampled: pd.DataFrame) -> pd.DataFrame:
frame = sampled.rename(columns=RENAME_MAP).copy()
frame["start_time"] = pd.to_datetime(frame["start_time"], errors="coerce")
numeric = [
"duration", "src_tos", "dst_tos", "total_packets", "total_bytes", "src_bytes"
]
if "dst_bytes" in frame.columns:
numeric.append("dst_bytes")
for column in numeric:
frame[column] = pd.to_numeric(frame[column], errors="coerce")
frame["src_port"] = parse_port(frame["src_port"])
frame["dst_port"] = parse_port(frame["dst_port"])
derived_dst_bytes = (frame["total_bytes"] - frame["src_bytes"]).clip(lower=0)
if "dst_bytes" in frame.columns:
frame["dst_bytes"] = frame["dst_bytes"].fillna(derived_dst_bytes).clip(lower=0)
else:
frame["dst_bytes"] = derived_dst_bytes
frame["bytes_per_packet"] = frame["total_bytes"] / frame["total_packets"].clip(lower=1)
frame["src_byte_ratio"] = frame["src_bytes"] / frame["total_bytes"].clip(lower=1)
frame["start_hour"] = frame["start_time"].dt.hour
frame["is_well_known_dst_port"] = frame["dst_port"].between(0, 1023).astype("Int64")
frame["binary_label"] = frame["label_family"].map(
{"Normal": 0, "Botnet": 1, "Background": -1}
).astype(int)
ordered = [
"start_time", "duration", "protocol", "src_addr", "src_port", "direction",
"dst_addr", "dst_port", "state", "src_tos", "dst_tos", "total_packets",
"total_bytes", "src_bytes", "dst_bytes", "bytes_per_packet", "src_byte_ratio",
"start_hour", "is_well_known_dst_port", "raw_label", "label_family", "binary_label",
]
return frame[ordered].sort_values("start_time", na_position="last").reset_index(drop=True)
def main() -> None:
args = parse_args()
input_path = find_input(args.input)
output_path = args.output.expanduser().resolve()
output_path.parent.mkdir(parents=True, exist_ok=True)
sampled = bounded_sample(input_path, args.per_class, args.chunk_size)
prepared = engineer_features(sampled)
prepared.to_csv(output_path, index=False)
print(f"Created {output_path} with shape {prepared.shape}")
print(prepared["label_family"].value_counts())
print("Background is retained for triage but excluded from supervised ground-truth metrics.")
if __name__ == "__main__":
main()
Lab-6/Lab-6-Solution/data/prepare_dataset.py
+123
View File
@@ -0,0 +1,123 @@
"""Prepare fixed NSL-KDD train/test tables for Module 6 notebooks."""
from __future__ import annotations
import argparse
from pathlib import Path
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
FEATURE_NAMES = [
"duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes",
"land", "wrong_fragment", "urgent", "hot", "num_failed_logins",
"logged_in", "num_compromised", "root_shell", "su_attempted", "num_root",
"num_file_creations", "num_shells", "num_access_files", "num_outbound_cmds",
"is_host_login", "is_guest_login", "count", "srv_count", "serror_rate",
"srv_serror_rate", "rerror_rate", "srv_rerror_rate", "same_srv_rate",
"diff_srv_rate", "srv_diff_host_rate", "dst_host_count",
"dst_host_srv_count", "dst_host_same_srv_rate", "dst_host_diff_srv_rate",
"dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate",
"dst_host_serror_rate", "dst_host_srv_serror_rate", "dst_host_rerror_rate",
"dst_host_srv_rerror_rate",
]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]
ATTACK_GROUPS = {
"normal": "Normal",
"back": "DoS", "land": "DoS", "neptune": "DoS", "pod": "DoS",
"smurf": "DoS", "teardrop": "DoS", "apache2": "DoS", "mailbomb": "DoS",
"processtable": "DoS", "udpstorm": "DoS", "worm": "DoS",
"ipsweep": "Probe", "nmap": "Probe", "portsweep": "Probe", "satan": "Probe",
"mscan": "Probe", "saint": "Probe",
"ftp_write": "R2L", "guess_passwd": "R2L", "imap": "R2L",
"multihop": "R2L", "phf": "R2L", "spy": "R2L", "warezclient": "R2L",
"warezmaster": "R2L", "sendmail": "R2L", "named": "R2L",
"snmpgetattack": "R2L", "snmpguess": "R2L", "xlock": "R2L",
"xsnoop": "R2L", "httptunnel": "R2L",
"buffer_overflow": "U2R", "loadmodule": "U2R", "perl": "U2R",
"rootkit": "U2R", "ps": "U2R", "sqlattack": "U2R", "xterm": "U2R",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--full-train",
action="store_true",
help="Use KDDTrain+.txt instead of KDDTrain+_20Percent.txt.",
)
return parser.parse_args()
def find_one(filename: str) -> Path:
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and verify the extracted files."
)
return matches[0]
def load_nsl_kdd(path: Path) -> pd.DataFrame:
columns = FEATURE_NAMES + ["attack_label", "difficulty"]
frame = pd.read_csv(path, header=None, names=columns, low_memory=False)
frame["attack_label"] = (
frame["attack_label"].astype(str).str.strip().str.rstrip(".").str.lower()
)
unknown = sorted(set(frame["attack_label"]) - set(ATTACK_GROUPS))
if unknown:
raise ValueError(f"Unknown attack labels in {path.name}: {unknown}")
for column in CATEGORICAL_FEATURES:
frame[column] = frame[column].astype(str).str.strip().str.lower()
numeric_features = [name for name in FEATURE_NAMES if name not in CATEGORICAL_FEATURES]
for column in numeric_features + ["difficulty"]:
frame[column] = pd.to_numeric(frame[column], errors="raise")
frame["attack_group"] = frame["attack_label"].map(ATTACK_GROUPS)
frame["binary_label"] = (frame["attack_group"] != "Normal").astype(int)
return frame[FEATURE_NAMES + ["attack_label", "attack_group", "binary_label", "difficulty"]]
def main() -> None:
args = parse_args()
train_name = "KDDTrain+.txt" if args.full_train else "KDDTrain+_20Percent.txt"
train = load_nsl_kdd(find_one(train_name))
test = load_nsl_kdd(find_one("KDDTest+.txt"))
train_path = HERE / "nsl_kdd_train.csv"
test_path = HERE / "nsl_kdd_test.csv"
manifest_path = HERE / "feature_manifest.csv"
train.to_csv(train_path, index=False)
test.to_csv(test_path, index=False)
manifest = pd.DataFrame({
"feature": FEATURE_NAMES,
"feature_type": [
"categorical" if name in CATEGORICAL_FEATURES else "numeric"
for name in FEATURE_NAMES
],
"model_feature": True,
})
manifest.to_csv(manifest_path, index=False)
print(f"Created {train_path} with shape {train.shape}")
print(f"Created {test_path} with shape {test.shape}")
print(f"Created {manifest_path}")
print("Train attack groups:")
print(train["attack_group"].value_counts())
print("Test attack groups:")
print(test["attack_group"].value_counts())
if __name__ == "__main__":
main()