{ "cells": [ { "cell_type": "markdown", "id": "cell-001", "metadata": {}, "source": [ "# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n", "\n", " \n", "**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n", "\n", "> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers." ] }, { "cell_type": "markdown", "id": "cell-002", "metadata": {}, "source": [ "## Mission\n", "\n", "Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n", "\n", "The workflow deliberately separates the three CTU-13 label families:\n", "\n", "- **Botnet:** verified traffic produced by an infected host;\n", "- **Normal:** verified traffic produced by known normal hosts;\n", "- **Background:** traffic whose status is not known with confidence.\n", "\n", "Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n", "\n", "**Required outputs**\n", "\n", "- `output/ctu13_botnet_metrics.csv`\n", "- `output/ctu13_source_triage.csv`\n", "- `output/ctu13_top_flows.csv`\n", "- `output/ctu13_confusion_matrix.png`\n", "- `output/ctu13_score_distribution.png`\n", "- `output/ctu13_analyst_summary.md`" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-003", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n", "OUTPUT_DIR = Path(\"output\")\n", "OUTPUT_DIR.mkdir(exist_ok=True)\n", "\n", "if not DATA_PATH.exists():\n", " raise FileNotFoundError(\n", " f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n", " \"python data/prepare_ctu13.py \"\n", " )\n", "\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.metrics import (\n", " ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n", " confusion_matrix, f1_score, precision_score, recall_score,\n", ")\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder\n", "\n", "RANDOM_STATE = 42" ] }, { "cell_type": "markdown", "id": "cell-004", "metadata": {}, "source": [ "## 1. Audit the prepared scenario\n", "\n", "Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-005", "metadata": {}, "outputs": [], "source": [ "flows = pd.read_csv(DATA_PATH, low_memory=False, parse_dates=[\"start_time\"])\n", "expected_families = {\"Botnet\", \"Normal\", \"Background\"}\n", "print(\"Shape:\", flows.shape)\n", "display(flows.head(3))\n", "display(flows[\"label_family\"].value_counts().rename(\"flows\").to_frame())\n", "print(\"Raw label examples:\")\n", "for value in flows[\"raw_label\"].dropna().drop_duplicates().head(8):\n", " print(\" -\", value)" ] }, { "cell_type": "markdown", "id": "cell-006", "metadata": {}, "source": [ "### Suggested answers\n", "\n", "1. Background means the traffic could not be confidently assigned to the verified Botnet or Normal hosts. It is uncertain, not confirmed benign.\n", "2. Addresses can identify scenario-specific infected hosts, while label fields directly encode the target. Either shortcut would inflate performance without learning behavior.\n", "3. The preparation utility reports retained counts. All three families are expected for the recommended scenario, although only Botnet and Normal enter supervised metrics." ] }, { "cell_type": "markdown", "id": "cell-007", "metadata": {}, "source": [ "## 2. Define behavioral features and create three data splits\n", "\n", "Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n", "\n", "The validation set selects an operating threshold. The test set remains untouched until the final evaluation." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-008", "metadata": {}, "outputs": [], "source": [ "categorical_features = [\"protocol\", \"direction\", \"state\"]\n", "numeric_features = [\n", " \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n", " \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n", " \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n", " \"is_well_known_dst_port\",\n", "]\n", "feature_columns = categorical_features + numeric_features\n", "\n", "\n", "labeled = flows.loc[flows[\"label_family\"].isin([\"Botnet\", \"Normal\"])].copy()\n", "background = flows.loc[flows[\"label_family\"].eq(\"Background\")].copy()\n", "X = labeled[feature_columns]\n", "y = labeled[\"binary_label\"].astype(int)\n", "\n", "X_train_valid, X_test, y_train_valid, y_test = train_test_split(\n", " X, y, test_size=0.20, stratify=y, random_state=RANDOM_STATE\n", ")\n", "X_train, X_valid, y_train, y_valid = train_test_split(\n", " X_train_valid, y_train_valid, test_size=0.25,\n", " stratify=y_train_valid, random_state=RANDOM_STATE,\n", ")\n", "\n", "print(\"Train / validation / test:\", len(X_train), len(X_valid), len(X_test))\n", "print(\"Background records reserved for triage:\", len(background))" ] }, { "cell_type": "markdown", "id": "cell-009", "metadata": {}, "source": [ "## 3. Train a bounded Random Forest\n", "\n", "The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-010", "metadata": {}, "outputs": [], "source": [ "preprocessor = ColumnTransformer([\n", " (\"categorical\", Pipeline([\n", " (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n", " (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n", " ]), categorical_features),\n", " (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n", "])\n", "\n", "detector = Pipeline([\n", " (\"preprocessor\", preprocessor),\n", " (\"model\", RandomForestClassifier(\n", " n_estimators=200,\n", " max_depth=12,\n", " min_samples_leaf=3,\n", " class_weight=\"balanced_subsample\",\n", " random_state=RANDOM_STATE,\n", " n_jobs=-1,\n", " )),\n", "])\n", "\n", "\n", "detector.fit(X_train, y_train)\n", "validation_scores = detector.predict_proba(X_valid)[:, 1]" ] }, { "cell_type": "markdown", "id": "cell-011", "metadata": {}, "source": [ "## 4. Select a validation threshold under an alert-quality constraint\n", "\n", "Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-012", "metadata": {}, "outputs": [], "source": [ "threshold_rows = []\n", "for threshold in np.linspace(0.05, 0.95, 91):\n", " predictions = (validation_scores >= threshold).astype(int)\n", " tn, fp, fn, tp = confusion_matrix(y_valid, predictions, labels=[0, 1]).ravel()\n", " threshold_rows.append({\n", " \"threshold\": threshold,\n", " \"botnet_recall\": recall_score(y_valid, predictions, zero_division=0),\n", " \"botnet_precision\": precision_score(y_valid, predictions, zero_division=0),\n", " \"botnet_f1\": f1_score(y_valid, predictions, zero_division=0),\n", " \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n", " \"alerts\": int(predictions.sum()),\n", " })\n", "\n", "threshold_table = pd.DataFrame(threshold_rows)\n", "acceptable = threshold_table.loc[threshold_table[\"false_positive_rate\"] <= 0.05]\n", "selection_pool = acceptable if not acceptable.empty else threshold_table\n", "selected_row = selection_pool.sort_values(\n", " [\"botnet_recall\", \"botnet_f1\", \"false_positive_rate\"],\n", " ascending=[False, False, True],\n", ").iloc[0]\n", "operating_threshold = float(selected_row[\"threshold\"])\n", "display(selected_row.to_frame(\"selected_value\"))" ] }, { "cell_type": "markdown", "id": "cell-013", "metadata": {}, "source": [ "## 5. Evaluate once on the held-out test set\n", "\n", "Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-014", "metadata": {}, "outputs": [], "source": [ "test_scores = detector.predict_proba(X_test)[:, 1]\n", "test_predictions = (test_scores >= operating_threshold).astype(int)\n", "tn, fp, fn, tp = confusion_matrix(y_test, test_predictions, labels=[0, 1]).ravel()\n", "metrics_table = pd.DataFrame([{\n", " \"threshold\": operating_threshold,\n", " \"accuracy\": accuracy_score(y_test, test_predictions),\n", " \"botnet_precision\": precision_score(y_test, test_predictions, zero_division=0),\n", " \"botnet_recall\": recall_score(y_test, test_predictions, zero_division=0),\n", " \"botnet_f1\": f1_score(y_test, test_predictions, zero_division=0),\n", " \"pr_auc\": average_precision_score(y_test, test_scores),\n", " \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n", " \"false_positives\": int(fp),\n", " \"false_negatives\": int(fn),\n", " \"alerts_per_10000\": 10000 * int(test_predictions.sum()) / len(y_test),\n", "}])\n", "display(metrics_table.round(4))\n", "\n", "fig, ax = plt.subplots(figsize=(6, 5))\n", "ConfusionMatrixDisplay.from_predictions(\n", " y_test, test_predictions, labels=[0, 1],\n", " display_labels=[\"Normal\", \"Botnet\"], cmap=\"Blues\", colorbar=False, ax=ax,\n", ")\n", "ax.set_title(\"CTU-13 Botnet detector — held-out labeled flows\")\n", "fig.tight_layout()\n", "fig.savefig(OUTPUT_DIR / \"ctu13_confusion_matrix.png\", dpi=160)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-015", "metadata": {}, "source": [ "## 6. Score all flows and create source-level triage\n", "\n", "Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n", "\n", "A high Background score is a lead for investigation, not proof that the flow is malicious." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-016", "metadata": {}, "outputs": [], "source": [ "all_scores = detector.predict_proba(flows[feature_columns])[:, 1]\n", "scored_flows = flows.copy()\n", "scored_flows[\"botnet_score\"] = all_scores\n", "scored_flows[\"botnet_alert\"] = (all_scores >= operating_threshold).astype(int)\n", "\n", "source_triage = (\n", " scored_flows.groupby(\"src_addr\", dropna=False)\n", " .agg(\n", " flow_count=(\"src_addr\", \"size\"),\n", " mean_botnet_score=(\"botnet_score\", \"mean\"),\n", " max_botnet_score=(\"botnet_score\", \"max\"),\n", " botnet_alerts=(\"botnet_alert\", \"sum\"),\n", " observed_botnet_labels=(\"binary_label\", lambda values: int((values == 1).sum())),\n", " uncertain_background_flows=(\"binary_label\", lambda values: int((values == -1).sum())),\n", " )\n", " .reset_index()\n", ")\n", "source_triage[\"alert_rate\"] = (\n", " source_triage[\"botnet_alerts\"] / source_triage[\"flow_count\"]\n", ")\n", "source_triage = source_triage.sort_values(\n", " [\"botnet_alerts\", \"max_botnet_score\", \"flow_count\"], ascending=False\n", ")\n", "\n", "evidence_columns = [\n", " \"start_time\", \"src_addr\", \"dst_addr\", \"protocol\", \"src_port\", \"dst_port\",\n", " \"duration\", \"total_packets\", \"total_bytes\", \"label_family\", \"raw_label\",\n", " \"botnet_score\", \"botnet_alert\",\n", "]\n", "top_flows = scored_flows.loc[scored_flows[\"botnet_alert\"].eq(1), evidence_columns].sort_values(\n", " \"botnet_score\", ascending=False\n", ").head(500)\n", "display(source_triage.head(10))\n", "display(top_flows.head(10))\n", "\n", "fig, ax = plt.subplots(figsize=(9, 5))\n", "for family, color in [(\"Normal\", \"tab:green\"), (\"Botnet\", \"tab:red\"), (\"Background\", \"tab:gray\")]:\n", " values = scored_flows.loc[scored_flows[\"label_family\"].eq(family), \"botnet_score\"]\n", " if not values.empty:\n", " ax.hist(values, bins=50, alpha=0.45, density=True, label=family, color=color)\n", "ax.axvline(operating_threshold, color=\"black\", linestyle=\"--\", label=\"Operating threshold\")\n", "ax.set_title(\"CTU-13 botnet-score distribution\")\n", "ax.set_xlabel(\"Predicted Botnet probability\")\n", "ax.set_ylabel(\"Density\")\n", "ax.legend()\n", "fig.tight_layout()\n", "fig.savefig(OUTPUT_DIR / \"ctu13_score_distribution.png\", dpi=160)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-017", "metadata": {}, "source": [ "## 7. Export evidence and write the analyst summary\n", "\n", "The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-018", "metadata": {}, "outputs": [], "source": [ "metrics_table.to_csv(OUTPUT_DIR / \"ctu13_botnet_metrics.csv\", index=False)\n", "source_triage.head(500).to_csv(OUTPUT_DIR / \"ctu13_source_triage.csv\", index=False)\n", "top_flows.to_csv(OUTPUT_DIR / \"ctu13_top_flows.csv\", index=False)\n", "\n", "result = metrics_table.iloc[0]\n", "background_alerts = int(\n", " scored_flows.loc[scored_flows[\"label_family\"].eq(\"Background\"), \"botnet_alert\"].sum()\n", ")\n", "analyst_summary = \"\\n\".join([\n", " \"# CTU-13 Botnet Flow Detection — Analyst Summary\", \"\",\n", " \"## Labeled-flow evaluation\", \"\",\n", " f\"- Operating threshold selected on validation data: {operating_threshold:.2f}\",\n", " f\"- Botnet precision: {result['botnet_precision']:.3f}\",\n", " f\"- Botnet recall: {result['botnet_recall']:.3f}\",\n", " f\"- Botnet F1: {result['botnet_f1']:.3f}\",\n", " f\"- PR-AUC: {result['pr_auc']:.3f}\",\n", " f\"- False-positive rate: {result['false_positive_rate']:.3f}\", \"\",\n", " \"## Triage\", \"\",\n", " f\"- Background flows above threshold: {background_alerts}\",\n", " f\"- Highest-priority source: {source_triage.iloc[0]['src_addr']}\",\n", " \"- Background is uncertain traffic; an alert is an investigation lead, not verified ground truth.\", \"\",\n", " \"## Limitations\", \"\",\n", " \"This experiment uses a bounded sample from one CTU-13 scenario and a random stratified split.\",\n", " \"Its result may be optimistic for the same botnet family and network environment.\",\n", " \"A stronger follow-up trains on one or more scenarios and tests on a different scenario.\",\n", " \"IP addresses were excluded from model inputs and used only for analyst aggregation.\",\n", "])\n", "(OUTPUT_DIR / \"ctu13_analyst_summary.md\").write_text(analyst_summary, encoding=\"utf-8\")\n", "print(analyst_summary)" ] }, { "cell_type": "markdown", "id": "cell-019", "metadata": {}, "source": [ "## Self-check" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-020", "metadata": {}, "outputs": [], "source": [ "expected_files = {\n", " \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n", " \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n", " \"ctu13_analyst_summary.md\",\n", "}\n", "assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n", "assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n", "assert set(y_test.unique()) == {0, 1}\n", "assert 0.0 <= operating_threshold <= 1.0\n", "assert len(test_scores) == len(y_test)\n", "assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n", "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n", "print(\"✓ Lab 7 completed\")" ] }, { "cell_type": "markdown", "id": "cell-021", "metadata": {}, "source": [ "## Suggested review checklist\n", "\n", "- Botnet and Normal provide supervised ground truth; Background remains uncertain.\n", "- IP addresses and target-derived fields are not model inputs.\n", "- Training, validation, and test records are separated before model fitting.\n", "- The validation set selects the operating threshold; test labels are used once.\n", "- Test reporting includes recall, precision, F1, PR-AUC, false-positive rate, and alert volume.\n", "- Source aggregation supports investigation without letting the model memorize infected hosts.\n", "- The report identifies the optimistic same-scenario split and recommends cross-scenario evaluation." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 5 }