{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n",
    "\n",
    "\n",
    "**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Mission\n",
    "\n",
    "Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n",
    "\n",
    "The workflow deliberately separates the three CTU-13 label families:\n",
    "\n",
    "- **Botnet:** verified traffic produced by an infected host;\n",
    "- **Normal:** verified traffic produced by known normal hosts;\n",
    "- **Background:** traffic whose status is not known with confidence.\n",
    "\n",
    "Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n",
    "\n",
    "**Required outputs**\n",
    "\n",
    "- `output/ctu13_botnet_metrics.csv`\n",
    "- `output/ctu13_source_triage.csv`\n",
    "- `output/ctu13_top_flows.csv`\n",
    "- `output/ctu13_confusion_matrix.png`\n",
    "- `output/ctu13_score_distribution.png`\n",
    "- `output/ctu13_analyst_summary.md`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n",
    "        \"python data/prepare_ctu13.py <path-to-scenario.binetflow>\"\n",
    "    )\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import (\n",
    "    ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n",
    "    confusion_matrix, f1_score, precision_score, recall_score,\n",
    ")\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "RANDOM_STATE = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-004",
   "metadata": {},
   "source": [
    "## 1. Audit the prepared scenario\n",
    "\n",
    "Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: Load DATA_PATH into flows and inspect its shape and label counts.\n",
    "flows = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-006",
   "metadata": {},
   "source": [
    "### Audit questions\n",
    "\n",
    "1. Why is Background not equivalent to Normal?\n",
    "2. Why would `src_addr`, `dst_addr`, `raw_label`, or `label_family` cause leakage if used as model inputs?\n",
    "3. Does this prepared sample contain all three label families?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Define behavioral features and create three data splits\n",
    "\n",
    "Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n",
    "\n",
    "The validation set selects an operating threshold. The test set remains untouched until the final evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = [\"protocol\", \"direction\", \"state\"]\n",
    "numeric_features = [\n",
    "    \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n",
    "    \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n",
    "    \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n",
    "    \"is_well_known_dst_port\",\n",
    "]\n",
    "feature_columns = categorical_features + numeric_features\n",
    "\n",
    "\n",
    "# TODO 2: Keep only Botnet and Normal rows in `labeled`.\n",
    "# TODO 3: Put Background rows in a separate `background` table.\n",
    "labeled = ...\n",
    "background = ...\n",
    "X = ...\n",
    "y = ...\n",
    "\n",
    "# TODO 4: Create stratified 60% train, 20% validation, and 20% test splits.\n",
    "X_train_valid, X_test, y_train_valid, y_test = ...\n",
    "X_train, X_valid, y_train, y_valid = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## 3. Train a bounded Random Forest\n",
    "\n",
    "The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-010",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "    (\"categorical\", Pipeline([\n",
    "        (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
    "        (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
    "    ]), categorical_features),\n",
    "    (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
    "])\n",
    "\n",
    "detector = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"model\", RandomForestClassifier(\n",
    "        n_estimators=200,\n",
    "        max_depth=12,\n",
    "        min_samples_leaf=3,\n",
    "        class_weight=\"balanced_subsample\",\n",
    "        random_state=RANDOM_STATE,\n",
    "        n_jobs=-1,\n",
    "    )),\n",
    "])\n",
    "\n",
    "\n",
    "# TODO 5: Fit the pipeline on the training split.\n",
    "# TODO 6: Calculate Botnet probabilities for the validation split.\n",
    "...\n",
    "validation_scores = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## 4. Select a validation threshold under an alert-quality constraint\n",
    "\n",
    "Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-012",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: Build threshold_table and select operating_threshold using the rule above.\n",
    "threshold_table = ...\n",
    "operating_threshold = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-013",
   "metadata": {},
   "source": [
    "## 5. Evaluate once on the held-out test set\n",
    "\n",
    "Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-014",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 8: Score the test split, apply operating_threshold, and create metrics_table.\n",
    "test_scores = ...\n",
    "test_predictions = ...\n",
    "metrics_table = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-015",
   "metadata": {},
   "source": [
    "## 6. Score all flows and create source-level triage\n",
    "\n",
    "Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n",
    "\n",
    "A high Background score is a lead for investigation, not proof that the flow is malicious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 9: Score every flow and add botnet_score and botnet_alert columns.\n",
    "# TODO 10: Aggregate flow count, score statistics, and alert count by src_addr.\n",
    "all_scores = ...\n",
    "source_triage = ...\n",
    "top_flows = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-017",
   "metadata": {},
   "source": [
    "## 7. Export evidence and write the analyst summary\n",
    "\n",
    "The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-018",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 11: Export the three CSV files, two figures, and analyst summary.\n",
    "analyst_summary = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-019",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-020",
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_files = {\n",
    "    \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n",
    "    \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n",
    "    \"ctu13_analyst_summary.md\",\n",
    "}\n",
    "assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n",
    "assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n",
    "assert set(y_test.unique()) == {0, 1}\n",
    "assert 0.0 <= operating_threshold <= 1.0\n",
    "assert len(test_scores) == len(y_test)\n",
    "assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n",
    "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
    "print(\"✓ Lab 7 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-021",
   "metadata": {},
   "source": [
    "## Submission checklist\n",
    "\n",
    "1. Run the notebook from top to bottom and pass the self-check.\n",
    "2. Submit the completed Lab notebook and six generated output files.\n",
    "3. Explain why Background was excluded from supervised metrics.\n",
    "4. Describe why IP addresses are evidence fields rather than model features.\n",
    "5. Identify the highest-priority source and cite at least two supporting flow indicators.\n",
    "6. Propose a cross-scenario test that would provide stronger evidence of generalization."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}