mikrotik/labs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Browse Source
This commit is contained in:
mikrotik
2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Lab-6/Lab-6-Practical/01_NSL_KDD_Flow_Audit_Lab.ipynb
+212
View File
@@ -0,0 +1,212 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 1: Auditing NSL-KDD Flow Records\n",
"\n",
" \n",
"**Topics:** network flows, fixed split, labels, attack groups, leakage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- distinguish network-flow features from targets and audit fields;\n",
"- preserve the predefined NSL-KDD train/test split;\n",
"- describe binary labels and the DoS, Probe, R2L, and U2R groups;\n",
"- identify attack labels that appear only in the test set;\n",
"- document imbalance, duplicates, and benchmark limitations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load and validate the fixed split"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the prepared train and test tables.\n",
"train_df = None\n",
"test_df = None\n",
"\n",
"# TODO 2: create feature_cols and numeric_features without target/audit columns.\n",
"feature_cols = None\n",
"numeric_features = None\n",
"\n",
"print(\"Train shape:\", None)\n",
"print(\"Test shape:\", None)\n",
"print(\"Model features:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit classes, groups, and unseen attack labels"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-007",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: calculate binary-label and attack-group counts for both splits.\n",
"train_binary = None\n",
"test_binary = None\n",
"train_groups = None\n",
"test_groups = None\n",
"\n",
"# TODO 4: find attack labels present in test but absent from training.\n",
"train_attack_labels = None\n",
"test_attack_labels = None\n",
"unseen_test_attacks = None\n",
"\n",
"display(pd.DataFrame({\"train\": train_binary, \"test\": test_binary}))\n",
"print(\"Attack labels seen only in test:\", unseen_test_attacks)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit quality and leakage risks"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-009",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: build a per-feature table of missing counts and unique counts.\n",
"quality_summary = None\n",
"\n",
"# TODO 6: count exact duplicate rows separately in train and test.\n",
"duplicate_counts = None\n",
"\n",
"# TODO 7: list target/audit columns that must never enter the model.\n",
"forbidden_model_columns = None\n",
"print(duplicate_counts)\n",
"print(\"Columns excluded from modeling:\", forbidden_model_columns)"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Benchmark note\n",
"\n",
"NSL-KDD is useful for learning a reproducible IDS workflow. Its fixed test set includes attacks that are absent from training, but it remains a legacy benchmark and does not represent modern production traffic."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert len(feature_cols) == 41\n",
"assert set(CATEGORICAL_FEATURES).issubset(feature_cols)\n",
"assert set(train_df[\"binary_label\"].unique()) == {0, 1}\n",
"assert set(test_df[\"attack_group\"].unique()).issubset({\"Normal\", \"DoS\", \"Probe\", \"R2L\", \"U2R\"})\n",
"assert len(unseen_test_attacks) > 0\n",
"assert set(feature_cols).isdisjoint(TARGET_COLUMNS)\n",
"assert quality_summary.shape[0] == 41\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why should the predefined train/test split be preserved?\n",
"2. Why must `difficulty` and derived label columns be excluded from X?\n",
"3. Why can binary accuracy hide failure on rare attack groups?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/02_Flow_Features_Preprocessing_Lab.ipynb
+205
View File
@@ -0,0 +1,205 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
"\n",
"\n",
"**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- interpret basic, content, traffic, and host-based flow features;\n",
"- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
"- compare normal and attack distributions;\n",
"- build separate preprocessing plans for distance models and tree models;\n",
"- fit encoders and scalers using training data only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load both splits and define feature_cols/numeric_features.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"\n",
"# TODO 2: create feature_families for basic, content, traffic, and host-based features.\n",
"feature_families = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect byte-count skew and apply `log1p`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create log_src_bytes and log_dst_bytes using np.log1p.\n",
"exploratory = None\n",
"\n",
"# TODO 4: calculate class medians and plot a sampled log-byte scatter chart.\n",
"byte_summary = None\n",
"sample = None\n",
"display(byte_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build preprocessing plans for two model families"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: build supervised_preprocessor with OneHotEncoder for the three\n",
"# categorical features and median imputation for numeric features.\n",
"supervised_preprocessor = None\n",
"\n",
"distance_features = [\n",
" \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
" \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
"]\n",
"# TODO 6: build distance_preprocessor with median imputation and StandardScaler.\n",
"distance_preprocessor = None\n",
"\n",
"# TODO 7: fit both preprocessors on training data only and record output shapes.\n",
"supervised_shape = None\n",
"distance_shape = None\n",
"print(supervised_shape, distance_shape)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Why two plans?\n",
"\n",
"Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert len(feature_cols) == 41\n",
"assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
"assert supervised_shape[0] == 100\n",
"assert supervised_shape[1] > len(feature_cols)\n",
"assert distance_shape == (100, len(distance_features))\n",
"assert all(feature in numeric_features for feature in distance_features)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is `log1p` preferable to dropping zero-byte flows?\n",
"2. Why do distance methods need scaling while Decision Trees usually do not?\n",
"3. What information leaks when a scaler or encoder is fitted on test data?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/03_kNN_Anomaly_Scoring_Lab.ipynb
+222
View File
@@ -0,0 +1,222 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 3: kNN Anomaly Scoring from Normal Traffic\n",
"\n",
"\n",
"**Topics:** normal-only baseline, nearest-neighbor distance, validation threshold, alert rate\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- train an anomaly baseline using normal traffic only;\n",
"- turn nearest-neighbor distance into an anomaly score;\n",
"- choose a threshold from held-out normal validation scores;\n",
"- evaluate attack recall and false-positive rate on the fixed test set;\n",
"- explain how threshold choice controls SOC alert volume."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.neighbors import NearestNeighbors\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, select src_bytes/dst_bytes, and split normal training\n",
"# records into 80% baseline fit and 20% validation.\n",
"train_df = None\n",
"test_df = None\n",
"byte_features = [\"src_bytes\", \"dst_bytes\"]\n",
"normal_fit, normal_valid = None, None\n",
"\n",
"# TODO 2: implement log_bytes, fit StandardScaler on normal_fit only,\n",
"# and transform fit, validation, and test records.\n",
"def log_bytes(frame):\n",
" pass\n",
"\n",
"scaler = None\n",
"X_fit = None\n",
"X_valid = None\n",
"X_test = None\n",
"y_test = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit the baseline and calculate anomaly scores"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: fit NearestNeighbors with k=5 on X_fit.\n",
"k = 5\n",
"knn = None\n",
"\n",
"# TODO 4: define the anomaly score as distance to the kth neighbor.\n",
"def knn_anomaly_score(matrix):\n",
" pass\n",
"\n",
"# TODO 5: select the 99th percentile of validation normal scores and\n",
"# evaluate thresholded predictions on the test set.\n",
"validation_scores = None\n",
"test_scores = None\n",
"threshold = None\n",
"test_predictions = None\n",
"knn_metrics = None\n",
"display(pd.Series(knn_metrics, name=\"kNN anomaly detector\"))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Visualize scores and threshold sensitivity"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: plot normal/attack score histograms and draw the threshold.\n",
"\n",
"# TODO 7: compare 0.95, 0.975, 0.99, and 0.995 normal-score quantiles.\n",
"sensitivity = None\n",
"display(sensitivity)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"assert set(train_df.loc[normal_fit.index, \"binary_label\"].unique()) == {0}\n",
"assert len(validation_scores) == len(normal_valid)\n",
"assert len(test_scores) == len(test_df)\n",
"assert threshold == np.quantile(validation_scores, 0.99)\n",
"assert set(test_predictions).issubset({0, 1})\n",
"assert sensitivity.shape[0] == 4\n",
"print(\"✓ Lab 3 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why should attacks be excluded from the baseline fit data?\n",
"2. Why is the threshold selected from normal validation scores rather than test labels?\n",
"3. How does lowering the threshold affect missed attacks and analyst workload?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/04_OneClass_IsolationForest_Lab.ipynb
+225
View File
@@ -0,0 +1,225 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 4: One-Class SVM and Isolation Forest\n",
"\n",
" \n",
"**Topics:** normal-only learning, anomaly scores, validation thresholds, PR-AUC, runtime\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- fit two normal-only novelty detectors;\n",
"- standardize features for One-Class SVM;\n",
"- derive a common anomaly-score convention;\n",
"- compare detectors with the same normal-validation alert budget;\n",
"- explain performance and scalability trade-offs."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.svm import OneClassSVM\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, define anomaly_features, split normal training rows,\n",
"# and cap the normal fit sample at 5,000 records.\n",
"train_df = None\n",
"test_df = None\n",
"anomaly_features = None\n",
"normal_fit = None\n",
"normal_valid = None\n",
"X_test = None\n",
"y_test = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit the two detectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: build StandardScaler pipelines for OneClassSVM and IsolationForest.\n",
"detectors = None\n",
"\n",
"# TODO 3: fit both detectors on normal_fit only.\n",
"print(\"Fitted detectors:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Apply a common validation-threshold policy"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: for each detector, negate decision_function so larger means\n",
"# more anomalous, choose the 99th normal-validation percentile, and evaluate.\n",
"detector_outputs = None\n",
"detector_comparison = None\n",
"display(detector_comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Compare attack-group recall"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: calculate recall separately for DoS, Probe, R2L, and U2R.\n",
"group_recall = None\n",
"display(group_recall)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(detectors) == {\"One-Class SVM\", \"Isolation Forest\"}\n",
"assert len(normal_fit) <= 5000\n",
"assert all(len(output[\"test_scores\"]) == len(test_df) for output in detector_outputs.values())\n",
"assert detector_comparison.shape[0] == 2\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert detector_comparison[\"false_positive_rate\"].between(0, 1).all()\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is One-Class SVM trained on a capped sample?\n",
"2. Why convert both model outputs to a common anomaly-score direction?\n",
"3. Why should per-group recall accompany aggregate attack recall?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/05_Supervised_Intrusion_Tree_Lab.ipynb
+228
View File
@@ -0,0 +1,228 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 5: A Supervised Intrusion Decision Tree\n",
"\n",
" \n",
"**Topics:** known-attack classification, fixed test set, tree rules, group recall\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- train a binary classifier from labeled normal and attack records;\n",
"- apply categorical encoding inside a pipeline;\n",
"- evaluate the fixed test set without tuning on it;\n",
"- inspect Decision Tree rules and feature importance;\n",
"- compare aggregate attack recall with group-level recall."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"from sklearn.tree import DecisionTreeClassifier, export_text\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the fixed split and create X_train/y_train/X_test/y_test.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"X_train, y_train, X_test, y_test = None, None, None, None\n",
"\n",
"# TODO 2: build a ColumnTransformer and a DecisionTree pipeline using\n",
"# max_depth=8, min_samples_leaf=5, class_weight='balanced', random_state=42.\n",
"preprocessor = None\n",
"tree_pipeline = None\n",
"\n",
"# TODO 3: fit the pipeline on the fixed training split."
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Evaluate binary and attack-group performance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: predict labels/probabilities and calculate security metrics plus PR-AUC.\n",
"tree_predictions = None\n",
"tree_probabilities = None\n",
"tree_metrics = None\n",
"display(pd.Series(tree_metrics, name=\"Decision Tree\"))\n",
"\n",
"# TODO 5: calculate recall for DoS, Probe, R2L, and U2R.\n",
"group_recall = None\n",
"display(group_recall)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Interpret the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: obtain transformed feature names, calculate top-12 impurity\n",
"# importances, export rules through depth 3, and plot the importance table.\n",
"fitted_preprocessor = None\n",
"fitted_tree = None\n",
"transformed_names = None\n",
"importance = None\n",
"top_importance = None\n",
"rule_text = None\n",
"print(rule_text)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"High performance on NSL-KDD does not establish production readiness. Tree rules can reflect benchmark artifacts, and attack-group recall can expose failures hidden by the binary score."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 8\n",
"assert len(tree_predictions) == len(test_df)\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. How does supervised learning differ from normal-only novelty detection?\n",
"2. Which attack group is hardest for the tree, and why might binary metrics hide it?\n",
"3. Why should exported tree rules not be treated as universal IDS signatures?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/06_Capstone_Hybrid_IDS_Lab.ipynb
+288
View File
@@ -0,0 +1,288 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 6: Capstone — Hybrid Network Intrusion Triage\n",
"\n",
"\n",
"**Topics:** classifier, anomaly score, hybrid policy, alert budget, SOC report\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Compare three alert policies on the fixed NSL-KDD test set:\n",
"\n",
"1. a supervised Decision Tree for known attack patterns;\n",
"2. a normal-only Isolation Forest for anomaly hunting;\n",
"3. a hybrid OR policy that alerts when either detector fires.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/hybrid_ids_metrics.csv`\n",
"- `output/hybrid_ids_group_recall.csv`\n",
"- `output/hybrid_ids_top_alerts.csv`\n",
"- `output/hybrid_ids_confusion_matrix.png`\n",
"- `output/hybrid_ids_score_distribution.png`\n",
"- `output/hybrid_ids_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the fixed split and build the supervised feature matrices.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"X_train, y_train, X_test, y_test = None, None, None, None\n",
"\n",
"# TODO 2: fit the bounded, balanced Decision Tree pipeline from Lab 5.\n",
"tree_model = None\n",
"tree_predictions = None\n",
"tree_probabilities = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit a normal-only Isolation Forest and set its alert budget"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: define anomaly_features, split normal training data, fit an\n",
"# Isolation Forest pipeline, and select the 99th validation percentile.\n",
"anomaly_features = None\n",
"normal_fit = None\n",
"normal_valid = None\n",
"anomaly_model = None\n",
"validation_scores = None\n",
"anomaly_scores = None\n",
"anomaly_threshold = None\n",
"anomaly_predictions = None"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Compare classifier, anomaly, and hybrid policies"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: create Classifier, Anomaly, and Hybrid_OR predictions/scores.\n",
"policy_predictions = None\n",
"policy_scores = None\n",
"\n",
"# TODO 5: build metrics_table and select the highest-recall policy with\n",
"# FPR <= 0.10, breaking ties by F1 then lower FPR.\n",
"metrics_table = None\n",
"selected_policy = None\n",
"selected_predictions = None\n",
"display(metrics_table)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Audit attack groups and create the alert evidence table"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: calculate per-group recall for every policy.\n",
"group_recall = None\n",
"\n",
"# TODO 7: build alert_table, derive alert_reason, and keep the top 200\n",
"# selected alerts ordered by anomaly score and tree probability.\n",
"alert_table = None\n",
"top_alerts = None\n",
"display(top_alerts)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export evidence and write the SOC recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: save a confusion matrix and anomaly-score distribution chart.\n",
"\n",
"# TODO 9: write an English SOC summary covering the selected policy,\n",
"# metrics, alert volume, detector roles, weak groups, and limitations.\n",
"analyst_summary = None\n",
"\n",
"# TODO 10: export the metrics, group recall, top alerts, and summary.\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"hybrid_ids_metrics.csv\", \"hybrid_ids_group_recall.csv\",\n",
" \"hybrid_ids_top_alerts.csv\", \"hybrid_ids_confusion_matrix.png\",\n",
" \"hybrid_ids_score_distribution.png\", \"hybrid_ids_analyst_summary.md\",\n",
"}\n",
"assert set(policy_predictions) == {\"Classifier\", \"Anomaly\", \"Hybrid_OR\"}\n",
"assert selected_policy in policy_predictions\n",
"assert len(alert_table) == len(test_df)\n",
"assert set(group_recall[\"attack_group\"]) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"NSL-KDD\" in analyst_summary\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom.\n",
"2. Confirm that all six output files exist.\n",
"3. Verify that the selected policy follows the documented rule.\n",
"4. Explain how anomaly-only alerts differ from classifier-only alerts.\n",
"5. Identify the weakest attack group and one dataset limitation."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/07_Botnet_Flow_Detection_CTU13_Lab.ipynb
+356
View File
@@ -0,0 +1,356 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n",
"\n",
"\n",
"**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n",
"\n",
"The workflow deliberately separates the three CTU-13 label families:\n",
"\n",
"- **Botnet:** verified traffic produced by an infected host;\n",
"- **Normal:** verified traffic produced by known normal hosts;\n",
"- **Background:** traffic whose status is not known with confidence.\n",
"\n",
"Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/ctu13_botnet_metrics.csv`\n",
"- `output/ctu13_source_triage.csv`\n",
"- `output/ctu13_top_flows.csv`\n",
"- `output/ctu13_confusion_matrix.png`\n",
"- `output/ctu13_score_distribution.png`\n",
"- `output/ctu13_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n",
" \"python data/prepare_ctu13.py <path-to-scenario.binetflow>\"\n",
" )\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n",
" confusion_matrix, f1_score, precision_score, recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"\n",
"RANDOM_STATE = 42"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Audit the prepared scenario\n",
"\n",
"Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: Load DATA_PATH into flows and inspect its shape and label counts.\n",
"flows = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"### Audit questions\n",
"\n",
"1. Why is Background not equivalent to Normal?\n",
"2. Why would `src_addr`, `dst_addr`, `raw_label`, or `label_family` cause leakage if used as model inputs?\n",
"3. Does this prepared sample contain all three label families?"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Define behavioral features and create three data splits\n",
"\n",
"Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n",
"\n",
"The validation set selects an operating threshold. The test set remains untouched until the final evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"categorical_features = [\"protocol\", \"direction\", \"state\"]\n",
"numeric_features = [\n",
" \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n",
" \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n",
" \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n",
" \"is_well_known_dst_port\",\n",
"]\n",
"feature_columns = categorical_features + numeric_features\n",
"\n",
"\n",
"# TODO 2: Keep only Botnet and Normal rows in `labeled`.\n",
"# TODO 3: Put Background rows in a separate `background` table.\n",
"labeled = ...\n",
"background = ...\n",
"X = ...\n",
"y = ...\n",
"\n",
"# TODO 4: Create stratified 60% train, 20% validation, and 20% test splits.\n",
"X_train_valid, X_test, y_train_valid, y_test = ...\n",
"X_train, X_valid, y_train, y_valid = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Train a bounded Random Forest\n",
"\n",
"The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"preprocessor = ColumnTransformer([\n",
" (\"categorical\", Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
" (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
" ]), categorical_features),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"\n",
"detector = Pipeline([\n",
" (\"preprocessor\", preprocessor),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=200,\n",
" max_depth=12,\n",
" min_samples_leaf=3,\n",
" class_weight=\"balanced_subsample\",\n",
" random_state=RANDOM_STATE,\n",
" n_jobs=-1,\n",
" )),\n",
"])\n",
"\n",
"\n",
"# TODO 5: Fit the pipeline on the training split.\n",
"# TODO 6: Calculate Botnet probabilities for the validation split.\n",
"...\n",
"validation_scores = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Select a validation threshold under an alert-quality constraint\n",
"\n",
"Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: Build threshold_table and select operating_threshold using the rule above.\n",
"threshold_table = ...\n",
"operating_threshold = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## 5. Evaluate once on the held-out test set\n",
"\n",
"Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: Score the test split, apply operating_threshold, and create metrics_table.\n",
"test_scores = ...\n",
"test_predictions = ...\n",
"metrics_table = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## 6. Score all flows and create source-level triage\n",
"\n",
"Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n",
"\n",
"A high Background score is a lead for investigation, not proof that the flow is malicious."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-016",
"metadata": {},
"outputs": [],
"source": [
"# TODO 9: Score every flow and add botnet_score and botnet_alert columns.\n",
"# TODO 10: Aggregate flow count, score statistics, and alert count by src_addr.\n",
"all_scores = ...\n",
"source_triage = ...\n",
"top_flows = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-017",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n",
"\n",
"The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-018",
"metadata": {},
"outputs": [],
"source": [
"# TODO 11: Export the three CSV files, two figures, and analyst summary.\n",
"analyst_summary = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-019",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-020",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n",
" \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n",
" \"ctu13_analyst_summary.md\",\n",
"}\n",
"assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n",
"assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n",
"assert set(y_test.unique()) == {0, 1}\n",
"assert 0.0 <= operating_threshold <= 1.0\n",
"assert len(test_scores) == len(y_test)\n",
"assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 7 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-021",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom and pass the self-check.\n",
"2. Submit the completed Lab notebook and six generated output files.\n",
"3. Explain why Background was excluded from supervised metrics.\n",
"4. Describe why IP addresses are evidence fields rather than model features.\n",
"5. Identify the highest-priority source and cite at least two supporting flow indicators.\n",
"6. Propose a cross-scenario test that would provide stronger evidence of generalization."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/data/README.md
+174
View File
@@ -0,0 +1,174 @@
# Dataset setup — NSL-KDD
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
## Download
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
```text
KDDTrain+_20Percent.txt
KDDTest+.txt
```
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
Place the files as follows:
```text
Practical-6/
└── data/
├── prepare_dataset.py
└── raw/
├── KDDTrain+_20Percent.txt
└── KDDTest+.txt
```
## Prepare the classroom tables
From `Practical-6`, run:
```bash
python data/prepare_dataset.py
```
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
```bash
python data/prepare_dataset.py --full-train
```
The utility:
- applies the official 41 feature names;
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
- validates numeric and categorical columns;
- rejects unknown attack labels so mapping errors remain visible;
- preserves the original fixed train/test split;
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
print("Train:", train.shape)
print("Test:", test.shape)
print(train["binary_label"].value_counts().sort_index())
print(test["attack_group"].value_counts())
```
Each prepared table must have 41 model features plus these four audit/target columns:
```text
attack_label, attack_group, binary_label, difficulty
```
## Citation and redistribution
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
Retain this citation when redistributing prepared classroom data.
## Important limitations
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
- Class prevalence does not represent a production SOC.
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
- The dataset has no dedicated botnet label.
---
# Dataset setup — CTU-13 for Lab 7
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
## Recommended download
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
```text
capture20110816-3.binetflow
```
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
## Placement and preparation
Place the `.binetflow` file below `data/raw/`:
```text
Practical-6/
└── data/
├── prepare_ctu13.py
└── raw/
└── capture20110816-3.binetflow
```
From `Practical-6`, run:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
```
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
```bash
python data/prepare_ctu13.py
```
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
```
It creates:
```text
data/ctu13_botnet_flows.csv
```
The loader accepts both common CTU-13 flow schemas:
- the compact 15-column export containing `Dir`;
- the extended Argus export containing directional counters such as
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
For the extended schema, the classroom `direction` feature is derived as
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
and destination packet counters. When available, the original `DstBytes` value
is used instead of reconstructing it from total and source bytes.
## Label policy
- `Botnet` receives `binary_label = 1`.
- `Normal` receives `binary_label = 0`.
- `Background` receives `binary_label = -1` because its status is uncertain.
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
## Safety and leakage controls
- Use only `.binetflow` flow records; do not download or execute malware samples.
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
- Raw label fields are excluded from model features.
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
## Citation
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
Lab-6/Lab-6-Practical/data/ctu13_botnet_flows.csv
+46128
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/feature_manifest.csv
+42
View File
@@ -0,0 +1,42 @@
feature,feature_type,model_feature
duration,numeric,True
protocol_type,categorical,True
service,categorical,True
flag,categorical,True
src_bytes,numeric,True
dst_bytes,numeric,True
land,numeric,True
wrong_fragment,numeric,True
urgent,numeric,True
hot,numeric,True
num_failed_logins,numeric,True
logged_in,numeric,True
num_compromised,numeric,True
root_shell,numeric,True
su_attempted,numeric,True
num_root,numeric,True
num_file_creations,numeric,True
num_shells,numeric,True
num_access_files,numeric,True
num_outbound_cmds,numeric,True
is_host_login,numeric,True
is_guest_login,numeric,True
count,numeric,True
srv_count,numeric,True
serror_rate,numeric,True
srv_serror_rate,numeric,True
rerror_rate,numeric,True
srv_rerror_rate,numeric,True
same_srv_rate,numeric,True
diff_srv_rate,numeric,True
srv_diff_host_rate,numeric,True
dst_host_count,numeric,True
dst_host_srv_count,numeric,True
dst_host_same_srv_rate,numeric,True
dst_host_diff_srv_rate,numeric,True
dst_host_same_src_port_rate,numeric,True
dst_host_srv_diff_host_rate,numeric,True
dst_host_serror_rate,numeric,True
dst_host_srv_serror_rate,numeric,True
dst_host_rerror_rate,numeric,True
dst_host_srv_rerror_rate,numeric,True
1 feature feature_type model_feature
2 duration numeric True
3 protocol_type categorical True
4 service categorical True
5 flag categorical True
6 src_bytes numeric True
7 dst_bytes numeric True
8 land numeric True
9 wrong_fragment numeric True
10 urgent numeric True
11 hot numeric True
12 num_failed_logins numeric True
13 logged_in numeric True
14 num_compromised numeric True
15 root_shell numeric True
16 su_attempted numeric True
17 num_root numeric True
18 num_file_creations numeric True
19 num_shells numeric True
20 num_access_files numeric True
21 num_outbound_cmds numeric True
22 is_host_login numeric True
23 is_guest_login numeric True
24 count numeric True
25 srv_count numeric True
26 serror_rate numeric True
27 srv_serror_rate numeric True
28 rerror_rate numeric True
29 srv_rerror_rate numeric True
30 same_srv_rate numeric True
31 diff_srv_rate numeric True
32 srv_diff_host_rate numeric True
33 dst_host_count numeric True
34 dst_host_srv_count numeric True
35 dst_host_same_srv_rate numeric True
36 dst_host_diff_srv_rate numeric True
37 dst_host_same_src_port_rate numeric True
38 dst_host_srv_diff_host_rate numeric True
39 dst_host_serror_rate numeric True
40 dst_host_srv_serror_rate numeric True
41 dst_host_rerror_rate numeric True
42 dst_host_srv_rerror_rate numeric True
Lab-6/Lab-6-Practical/data/nsl_kdd_test.csv
+22545
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/nsl_kdd_train.csv
+125974
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/prepare_ctu13.py
+261
View File
@@ -0,0 +1,261 @@
"""Create a bounded, classroom-ready flow table from one CTU-13 .binetflow file."""
from __future__ import annotations
import argparse
from pathlib import Path
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
REQUIRED_RAW_COLUMNS = [
"StartTime", "Dur", "Proto", "SrcAddr", "Sport", "DstAddr", "Dport",
"State", "sTos", "dTos", "TotPkts", "TotBytes", "SrcBytes", "Label",
]
# CTU-13 is distributed in more than one Argus text schema. The compact
# 15-column export contains ``Dir``. The extended export supplied with some
# scenarios omits it but provides per-direction packet and byte counters.
OPTIONAL_RAW_COLUMNS = ["Dir", "SrcPkts", "DstPkts", "DstBytes"]
RENAME_MAP = {
"StartTime": "start_time",
"Dur": "duration",
"Proto": "protocol",
"SrcAddr": "src_addr",
"Sport": "src_port",
"Dir": "direction",
"DstAddr": "dst_addr",
"Dport": "dst_port",
"State": "state",
"sTos": "src_tos",
"dTos": "dst_tos",
"TotPkts": "total_packets",
"TotBytes": "total_bytes",
"SrcBytes": "src_bytes",
"DstBytes": "dst_bytes",
"Label": "raw_label",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"input",
nargs="?",
type=Path,
help="Path to one labeled CTU-13 .binetflow file. If omitted, data/raw is searched.",
)
parser.add_argument(
"--per-class",
type=int,
default=20_000,
help="Maximum sampled rows for each of Botnet, Normal, and Background.",
)
parser.add_argument(
"--chunk-size",
type=int,
default=200_000,
help="Rows read at a time; reduce this value on low-memory computers.",
)
parser.add_argument(
"--output",
type=Path,
default=HERE / "ctu13_botnet_flows.csv",
help="Prepared CSV destination.",
)
return parser.parse_args()
def find_input(requested: Path | None) -> Path:
if requested is not None:
path = requested.expanduser().resolve()
if not path.is_file():
raise FileNotFoundError(f"CTU-13 input file not found: {path}")
return path
matches = sorted(RAW_DIR.rglob("*.binetflow")) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one .binetflow file below {RAW_DIR}, found {len(matches)}. "
"Pass the input path explicitly when more than one file is present."
)
return matches[0]
def label_family(labels: pd.Series) -> pd.Series:
text = labels.fillna("").astype(str)
return pd.Series(
np.select(
[
text.str.contains("Botnet", case=False, regex=False),
text.str.contains("Normal", case=False, regex=False),
text.str.contains("Background", case=False, regex=False),
],
["Botnet", "Normal", "Background"],
default="Other",
),
index=labels.index,
dtype="string",
)
def parse_port(values: pd.Series) -> pd.Series:
text = values.fillna("").astype(str).str.strip().str.lower()
decimal = pd.to_numeric(text.where(~text.str.startswith("0x")), errors="coerce")
hexadecimal = text.where(text.str.startswith("0x")).map(
lambda value: int(value, 16) if isinstance(value, str) and value else np.nan
)
return decimal.fillna(hexadecimal).astype("Float64")
def inspect_schema(path: Path) -> tuple[list[str], str]:
"""Validate a compact or extended CTU-13 header and select needed fields."""
header = pd.read_csv(path, nrows=0).columns.tolist()
missing = sorted(set(REQUIRED_RAW_COLUMNS) - set(header))
if missing:
raise ValueError(
f"Missing required CTU-13 columns: {missing}. Available columns: {header}"
)
selected = REQUIRED_RAW_COLUMNS + [
column for column in OPTIONAL_RAW_COLUMNS if column in header
]
if "Dir" in header:
direction_mode = "source Dir column"
elif {"SrcPkts", "DstPkts"}.issubset(header):
direction_mode = "derived from SrcPkts/DstPkts"
else:
direction_mode = "constant unknown (no Dir or packet-direction counters)"
print(f"Detected {len(header)}-column CTU-13 schema; direction is {direction_mode}.")
return selected, direction_mode
def ensure_direction(frame: pd.DataFrame) -> pd.DataFrame:
"""Return a frame with a categorical Dir field for either CTU-13 schema."""
if "Dir" in frame.columns:
frame["Dir"] = frame["Dir"].fillna("unknown").astype(str).str.strip()
frame.loc[frame["Dir"].eq(""), "Dir"] = "unknown"
return frame
if {"SrcPkts", "DstPkts"}.issubset(frame.columns):
src_packets = pd.to_numeric(frame["SrcPkts"], errors="coerce").fillna(0)
dst_packets = pd.to_numeric(frame["DstPkts"], errors="coerce").fillna(0)
frame["Dir"] = np.select(
[
(src_packets > 0) & (dst_packets > 0),
(src_packets > 0) & (dst_packets <= 0),
(src_packets <= 0) & (dst_packets > 0),
],
["bidirectional", "source_only", "destination_only"],
default="unknown",
)
else:
frame["Dir"] = "unknown"
return frame
def bounded_sample(path: Path, per_class: int, chunk_size: int) -> pd.DataFrame:
if per_class < 1 or chunk_size < 1:
raise ValueError("--per-class and --chunk-size must be positive integers.")
kept = {family: pd.DataFrame() for family in ("Botnet", "Normal", "Background")}
rng = np.random.default_rng(42)
saw_header = False
selected_columns, _ = inspect_schema(path)
for chunk_number, chunk in enumerate(
pd.read_csv(
path,
usecols=selected_columns,
chunksize=chunk_size,
low_memory=False,
),
start=1,
):
saw_header = True
chunk = ensure_direction(chunk)
output_columns = REQUIRED_RAW_COLUMNS + ["Dir"]
if "DstBytes" in chunk.columns:
output_columns.append("DstBytes")
chunk = chunk[output_columns].copy()
chunk["label_family"] = label_family(chunk["Label"])
for family in kept:
part = chunk.loc[chunk["label_family"].eq(family)].copy()
if part.empty:
continue
part["_sample_key"] = rng.random(len(part))
candidates = pd.concat([kept[family], part], ignore_index=True)
kept[family] = candidates.nsmallest(per_class, "_sample_key")
counts = {family: len(frame) for family, frame in kept.items()}
print(f"Processed chunk {chunk_number}: retained {counts}")
if not saw_header:
raise ValueError(f"No rows found in {path}")
sampled = pd.concat(kept.values(), ignore_index=True).drop(columns="_sample_key")
if sampled.empty:
raise ValueError("No Botnet, Normal, or Background labels were found.")
return sampled
def engineer_features(sampled: pd.DataFrame) -> pd.DataFrame:
frame = sampled.rename(columns=RENAME_MAP).copy()
frame["start_time"] = pd.to_datetime(frame["start_time"], errors="coerce")
numeric = [
"duration", "src_tos", "dst_tos", "total_packets", "total_bytes", "src_bytes"
]
if "dst_bytes" in frame.columns:
numeric.append("dst_bytes")
for column in numeric:
frame[column] = pd.to_numeric(frame[column], errors="coerce")
frame["src_port"] = parse_port(frame["src_port"])
frame["dst_port"] = parse_port(frame["dst_port"])
derived_dst_bytes = (frame["total_bytes"] - frame["src_bytes"]).clip(lower=0)
if "dst_bytes" in frame.columns:
frame["dst_bytes"] = frame["dst_bytes"].fillna(derived_dst_bytes).clip(lower=0)
else:
frame["dst_bytes"] = derived_dst_bytes
frame["bytes_per_packet"] = frame["total_bytes"] / frame["total_packets"].clip(lower=1)
frame["src_byte_ratio"] = frame["src_bytes"] / frame["total_bytes"].clip(lower=1)
frame["start_hour"] = frame["start_time"].dt.hour
frame["is_well_known_dst_port"] = frame["dst_port"].between(0, 1023).astype("Int64")
frame["binary_label"] = frame["label_family"].map(
{"Normal": 0, "Botnet": 1, "Background": -1}
).astype(int)
ordered = [
"start_time", "duration", "protocol", "src_addr", "src_port", "direction",
"dst_addr", "dst_port", "state", "src_tos", "dst_tos", "total_packets",
"total_bytes", "src_bytes", "dst_bytes", "bytes_per_packet", "src_byte_ratio",
"start_hour", "is_well_known_dst_port", "raw_label", "label_family", "binary_label",
]
return frame[ordered].sort_values("start_time", na_position="last").reset_index(drop=True)
def main() -> None:
args = parse_args()
input_path = find_input(args.input)
output_path = args.output.expanduser().resolve()
output_path.parent.mkdir(parents=True, exist_ok=True)
sampled = bounded_sample(input_path, args.per_class, args.chunk_size)
prepared = engineer_features(sampled)
prepared.to_csv(output_path, index=False)
print(f"Created {output_path} with shape {prepared.shape}")
print(prepared["label_family"].value_counts())
print("Background is retained for triage but excluded from supervised ground-truth metrics.")
if __name__ == "__main__":
main()
Lab-6/Lab-6-Practical/data/prepare_dataset.py
+123
View File
@@ -0,0 +1,123 @@
"""Prepare fixed NSL-KDD train/test tables for Module 6 notebooks."""
from __future__ import annotations
import argparse
from pathlib import Path
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
FEATURE_NAMES = [
"duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes",
"land", "wrong_fragment", "urgent", "hot", "num_failed_logins",
"logged_in", "num_compromised", "root_shell", "su_attempted", "num_root",
"num_file_creations", "num_shells", "num_access_files", "num_outbound_cmds",
"is_host_login", "is_guest_login", "count", "srv_count", "serror_rate",
"srv_serror_rate", "rerror_rate", "srv_rerror_rate", "same_srv_rate",
"diff_srv_rate", "srv_diff_host_rate", "dst_host_count",
"dst_host_srv_count", "dst_host_same_srv_rate", "dst_host_diff_srv_rate",
"dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate",
"dst_host_serror_rate", "dst_host_srv_serror_rate", "dst_host_rerror_rate",
"dst_host_srv_rerror_rate",
]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]
ATTACK_GROUPS = {
"normal": "Normal",
"back": "DoS", "land": "DoS", "neptune": "DoS", "pod": "DoS",
"smurf": "DoS", "teardrop": "DoS", "apache2": "DoS", "mailbomb": "DoS",
"processtable": "DoS", "udpstorm": "DoS", "worm": "DoS",
"ipsweep": "Probe", "nmap": "Probe", "portsweep": "Probe", "satan": "Probe",
"mscan": "Probe", "saint": "Probe",
"ftp_write": "R2L", "guess_passwd": "R2L", "imap": "R2L",
"multihop": "R2L", "phf": "R2L", "spy": "R2L", "warezclient": "R2L",
"warezmaster": "R2L", "sendmail": "R2L", "named": "R2L",
"snmpgetattack": "R2L", "snmpguess": "R2L", "xlock": "R2L",
"xsnoop": "R2L", "httptunnel": "R2L",
"buffer_overflow": "U2R", "loadmodule": "U2R", "perl": "U2R",
"rootkit": "U2R", "ps": "U2R", "sqlattack": "U2R", "xterm": "U2R",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--full-train",
action="store_true",
help="Use KDDTrain+.txt instead of KDDTrain+_20Percent.txt.",
)
return parser.parse_args()
def find_one(filename: str) -> Path:
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and verify the extracted files."
)
return matches[0]
def load_nsl_kdd(path: Path) -> pd.DataFrame:
columns = FEATURE_NAMES + ["attack_label", "difficulty"]
frame = pd.read_csv(path, header=None, names=columns, low_memory=False)
frame["attack_label"] = (
frame["attack_label"].astype(str).str.strip().str.rstrip(".").str.lower()
)
unknown = sorted(set(frame["attack_label"]) - set(ATTACK_GROUPS))
if unknown:
raise ValueError(f"Unknown attack labels in {path.name}: {unknown}")
for column in CATEGORICAL_FEATURES:
frame[column] = frame[column].astype(str).str.strip().str.lower()
numeric_features = [name for name in FEATURE_NAMES if name not in CATEGORICAL_FEATURES]
for column in numeric_features + ["difficulty"]:
frame[column] = pd.to_numeric(frame[column], errors="raise")
frame["attack_group"] = frame["attack_label"].map(ATTACK_GROUPS)
frame["binary_label"] = (frame["attack_group"] != "Normal").astype(int)
return frame[FEATURE_NAMES + ["attack_label", "attack_group", "binary_label", "difficulty"]]
def main() -> None:
args = parse_args()
train_name = "KDDTrain+.txt" if args.full_train else "KDDTrain+_20Percent.txt"
train = load_nsl_kdd(find_one(train_name))
test = load_nsl_kdd(find_one("KDDTest+.txt"))
train_path = HERE / "nsl_kdd_train.csv"
test_path = HERE / "nsl_kdd_test.csv"
manifest_path = HERE / "feature_manifest.csv"
train.to_csv(train_path, index=False)
test.to_csv(test_path, index=False)
manifest = pd.DataFrame({
"feature": FEATURE_NAMES,
"feature_type": [
"categorical" if name in CATEGORICAL_FEATURES else "numeric"
for name in FEATURE_NAMES
],
"model_feature": True,
})
manifest.to_csv(manifest_path, index=False)
print(f"Created {train_path} with shape {train.shape}")
print(f"Created {test_path} with shape {test.shape}")
print(f"Created {manifest_path}")
print("Train attack groups:")
print(train["attack_group"].value_counts())
print("Test attack groups:")
print(test["attack_group"].value_counts())
if __name__ == "__main__":
main()