labs/Lab-6/Lab-6-Practical/07_Botnet_Flow_Detection_CTU13_Lab.ipynb

Module 6 — Lab 7: Botnet Flow Detection with CTU-13

Topics: CTU-13, flow labels, botnet classification, threshold selection, source triage

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Mission

Build a lightweight botnet detector from one labeled CTU-13 .binetflow scenario.

The workflow deliberately separates the three CTU-13 label families:

• Botnet: verified traffic produced by an infected host;
• Normal: verified traffic produced by known normal hosts;
• Background: traffic whose status is not known with confidence.

Train and evaluate only on Botnet versus Normal records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.

Required outputs

• output/ctu13_botnet_metrics.csv
• output/ctu13_source_triage.csv
• output/ctu13_top_flows.csv
• output/ctu13_confusion_matrix.png
• output/ctu13_score_distribution.png
• output/ctu13_analyst_summary.md

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/ctu13_botnet_flows.csv")
OUTPUT_DIR = Path("output")
OUTPUT_DIR.mkdir(exist_ok=True)

if not DATA_PATH.exists():
    raise FileNotFoundError(
        f"Missing {DATA_PATH}. Follow data/README.md and run "
        "python data/prepare_ctu13.py <path-to-scenario.binetflow>"
    )

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import (
    ConfusionMatrixDisplay, accuracy_score, average_precision_score,
    confusion_matrix, f1_score, precision_score, recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

RANDOM_STATE = 42

1. Audit the prepared scenario

Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior.

In [ ]:

# TODO 1: Load DATA_PATH into flows and inspect its shape and label counts.
flows = ...

Audit questions

1. Why is Background not equivalent to Normal?
2. Why would src_addr, dst_addr, raw_label, or label_family cause leakage if used as model inputs?
3. Does this prepared sample contain all three label families?

2. Define behavioral features and create three data splits

Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.

The validation set selects an operating threshold. The test set remains untouched until the final evaluation.

In [ ]:

categorical_features = ["protocol", "direction", "state"]
numeric_features = [
    "duration", "src_port", "dst_port", "src_tos", "dst_tos",
    "total_packets", "total_bytes", "src_bytes", "dst_bytes",
    "bytes_per_packet", "src_byte_ratio", "start_hour",
    "is_well_known_dst_port",
]
feature_columns = categorical_features + numeric_features


# TODO 2: Keep only Botnet and Normal rows in `labeled`.
# TODO 3: Put Background rows in a separate `background` table.
labeled = ...
background = ...
X = ...
y = ...

# TODO 4: Create stratified 60% train, 20% validation, and 20% test splits.
X_train_valid, X_test, y_train_valid, y_test = ...
X_train, X_valid, y_train, y_valid = ...

3. Train a bounded Random Forest

The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer.

In [ ]:

preprocessor = ColumnTransformer([
    ("categorical", Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("encoder", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_features),
    ("numeric", SimpleImputer(strategy="median"), numeric_features),
])

detector = Pipeline([
    ("preprocessor", preprocessor),
    ("model", RandomForestClassifier(
        n_estimators=200,
        max_depth=12,
        min_samples_leaf=3,
        class_weight="balanced_subsample",
        random_state=RANDOM_STATE,
        n_jobs=-1,
    )),
])


# TODO 5: Fit the pipeline on the training split.
# TODO 6: Calculate Botnet probabilities for the validation split.
...
validation_scores = ...

4. Select a validation threshold under an alert-quality constraint

Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold.

In [ ]:

# TODO 7: Build threshold_table and select operating_threshold using the rule above.
threshold_table = ...
operating_threshold = ...

5. Evaluate once on the held-out test set

Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context.

In [ ]:

# TODO 8: Score the test split, apply operating_threshold, and create metrics_table.
test_scores = ...
test_predictions = ...
metrics_table = ...

6. Score all flows and create source-level triage

Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.

A high Background score is a lead for investigation, not proof that the flow is malicious.

In [ ]:

# TODO 9: Score every flow and add botnet_score and botnet_alert columns.
# TODO 10: Aggregate flow count, score statistics, and alert count by src_addr.
all_scores = ...
source_triage = ...
top_flows = ...

7. Export evidence and write the analyst summary

The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic.

In [ ]:

# TODO 11: Export the three CSV files, two figures, and analyst summary.
analyst_summary = ...

Self-check

In [ ]:

expected_files = {
    "ctu13_botnet_metrics.csv", "ctu13_source_triage.csv", "ctu13_top_flows.csv",
    "ctu13_confusion_matrix.png", "ctu13_score_distribution.png",
    "ctu13_analyst_summary.md",
}
assert set(flows["label_family"]).issubset({"Botnet", "Normal", "Background"})
assert not {"src_addr", "dst_addr", "raw_label", "label_family", "binary_label"}.intersection(feature_columns)
assert set(y_test.unique()) == {0, 1}
assert 0.0 <= operating_threshold <= 1.0
assert len(test_scores) == len(y_test)
assert set(source_triage.columns) >= {"src_addr", "flow_count", "botnet_alerts", "alert_rate"}
assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})
print("✓ Lab 7 completed")

Submission checklist

1. Run the notebook from top to bottom and pass the self-check.
2. Submit the completed Lab notebook and six generated output files.
3. Explain why Background was excluded from supervised metrics.
4. Describe why IP addresses are evidence fields rather than model features.
5. Identify the highest-priority source and cite at least two supporting flow indicators.
6. Propose a cross-scenario test that would provide stronger evidence of generalization.