CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,356 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-001",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n",
|
||||
"\n",
|
||||
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-002",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Mission\n",
|
||||
"\n",
|
||||
"Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n",
|
||||
"\n",
|
||||
"The workflow deliberately separates the three CTU-13 label families:\n",
|
||||
"\n",
|
||||
"- **Botnet:** verified traffic produced by an infected host;\n",
|
||||
"- **Normal:** verified traffic produced by known normal hosts;\n",
|
||||
"- **Background:** traffic whose status is not known with confidence.\n",
|
||||
"\n",
|
||||
"Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n",
|
||||
"\n",
|
||||
"**Required outputs**\n",
|
||||
"\n",
|
||||
"- `output/ctu13_botnet_metrics.csv`\n",
|
||||
"- `output/ctu13_source_triage.csv`\n",
|
||||
"- `output/ctu13_top_flows.csv`\n",
|
||||
"- `output/ctu13_confusion_matrix.png`\n",
|
||||
"- `output/ctu13_score_distribution.png`\n",
|
||||
"- `output/ctu13_analyst_summary.md`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-003",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n",
|
||||
"OUTPUT_DIR = Path(\"output\")\n",
|
||||
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n",
|
||||
" \"python data/prepare_ctu13.py <path-to-scenario.binetflow>\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"from sklearn.compose import ColumnTransformer\n",
|
||||
"from sklearn.ensemble import RandomForestClassifier\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.metrics import (\n",
|
||||
" ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n",
|
||||
" confusion_matrix, f1_score, precision_score, recall_score,\n",
|
||||
")\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.preprocessing import OneHotEncoder\n",
|
||||
"\n",
|
||||
"RANDOM_STATE = 42"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-004",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Audit the prepared scenario\n",
|
||||
"\n",
|
||||
"Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-005",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 1: Load DATA_PATH into flows and inspect its shape and label counts.\n",
|
||||
"flows = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-006",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Audit questions\n",
|
||||
"\n",
|
||||
"1. Why is Background not equivalent to Normal?\n",
|
||||
"2. Why would `src_addr`, `dst_addr`, `raw_label`, or `label_family` cause leakage if used as model inputs?\n",
|
||||
"3. Does this prepared sample contain all three label families?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-007",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Define behavioral features and create three data splits\n",
|
||||
"\n",
|
||||
"Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n",
|
||||
"\n",
|
||||
"The validation set selects an operating threshold. The test set remains untouched until the final evaluation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-008",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"categorical_features = [\"protocol\", \"direction\", \"state\"]\n",
|
||||
"numeric_features = [\n",
|
||||
" \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n",
|
||||
" \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n",
|
||||
" \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n",
|
||||
" \"is_well_known_dst_port\",\n",
|
||||
"]\n",
|
||||
"feature_columns = categorical_features + numeric_features\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# TODO 2: Keep only Botnet and Normal rows in `labeled`.\n",
|
||||
"# TODO 3: Put Background rows in a separate `background` table.\n",
|
||||
"labeled = ...\n",
|
||||
"background = ...\n",
|
||||
"X = ...\n",
|
||||
"y = ...\n",
|
||||
"\n",
|
||||
"# TODO 4: Create stratified 60% train, 20% validation, and 20% test splits.\n",
|
||||
"X_train_valid, X_test, y_train_valid, y_test = ...\n",
|
||||
"X_train, X_valid, y_train, y_valid = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-009",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Train a bounded Random Forest\n",
|
||||
"\n",
|
||||
"The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-010",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"preprocessor = ColumnTransformer([\n",
|
||||
" (\"categorical\", Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
|
||||
" (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
|
||||
" ]), categorical_features),\n",
|
||||
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"detector = Pipeline([\n",
|
||||
" (\"preprocessor\", preprocessor),\n",
|
||||
" (\"model\", RandomForestClassifier(\n",
|
||||
" n_estimators=200,\n",
|
||||
" max_depth=12,\n",
|
||||
" min_samples_leaf=3,\n",
|
||||
" class_weight=\"balanced_subsample\",\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" n_jobs=-1,\n",
|
||||
" )),\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# TODO 5: Fit the pipeline on the training split.\n",
|
||||
"# TODO 6: Calculate Botnet probabilities for the validation split.\n",
|
||||
"...\n",
|
||||
"validation_scores = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-011",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Select a validation threshold under an alert-quality constraint\n",
|
||||
"\n",
|
||||
"Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-012",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 7: Build threshold_table and select operating_threshold using the rule above.\n",
|
||||
"threshold_table = ...\n",
|
||||
"operating_threshold = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-013",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. Evaluate once on the held-out test set\n",
|
||||
"\n",
|
||||
"Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-014",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 8: Score the test split, apply operating_threshold, and create metrics_table.\n",
|
||||
"test_scores = ...\n",
|
||||
"test_predictions = ...\n",
|
||||
"metrics_table = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-015",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 6. Score all flows and create source-level triage\n",
|
||||
"\n",
|
||||
"Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n",
|
||||
"\n",
|
||||
"A high Background score is a lead for investigation, not proof that the flow is malicious."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-016",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 9: Score every flow and add botnet_score and botnet_alert columns.\n",
|
||||
"# TODO 10: Aggregate flow count, score statistics, and alert count by src_addr.\n",
|
||||
"all_scores = ...\n",
|
||||
"source_triage = ...\n",
|
||||
"top_flows = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-017",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 7. Export evidence and write the analyst summary\n",
|
||||
"\n",
|
||||
"The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-018",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 11: Export the three CSV files, two figures, and analyst summary.\n",
|
||||
"analyst_summary = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-019",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-020",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"expected_files = {\n",
|
||||
" \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n",
|
||||
" \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n",
|
||||
" \"ctu13_analyst_summary.md\",\n",
|
||||
"}\n",
|
||||
"assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n",
|
||||
"assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n",
|
||||
"assert set(y_test.unique()) == {0, 1}\n",
|
||||
"assert 0.0 <= operating_threshold <= 1.0\n",
|
||||
"assert len(test_scores) == len(y_test)\n",
|
||||
"assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n",
|
||||
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
|
||||
"print(\"✓ Lab 7 completed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-021",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Submission checklist\n",
|
||||
"\n",
|
||||
"1. Run the notebook from top to bottom and pass the self-check.\n",
|
||||
"2. Submit the completed Lab notebook and six generated output files.\n",
|
||||
"3. Explain why Background was excluded from supervised metrics.\n",
|
||||
"4. Describe why IP addresses are evidence fields rather than model features.\n",
|
||||
"5. Identify the highest-priority source and cite at least two supporting flow indicators.\n",
|
||||
"6. Propose a cross-scenario test that would provide stronger evidence of generalization."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user