labs/Lab-6/Lab-6-Practical/06_Capstone_Hybrid_IDS_Lab.ipynb

Module 6 — Lab 6: Capstone — Hybrid Network Intrusion Triage

Topics: classifier, anomaly score, hybrid policy, alert budget, SOC report

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Mission

Compare three alert policies on the fixed NSL-KDD test set:

1. a supervised Decision Tree for known attack patterns;
2. a normal-only Isolation Forest for anomaly hunting;
3. a hybrid OR policy that alerts when either detector fires.

Required outputs

• output/hybrid_ids_metrics.csv
• output/hybrid_ids_group_recall.csv
• output/hybrid_ids_top_alerts.csv
• output/hybrid_ids_confusion_matrix.png
• output/hybrid_ids_score_distribution.png
• output/hybrid_ids_analyst_summary.md

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

TRAIN_PATH = Path("data/nsl_kdd_train.csv")
TEST_PATH = Path("data/nsl_kdd_test.csv")

missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]
if missing_files:
    raise FileNotFoundError(
        "Missing prepared NSL-KDD files: " + ", ".join(missing_files) +
        ". Follow data/README.md and run: python data/prepare_dataset.py"
    )

TARGET_COLUMNS = ["attack_label", "attack_group", "binary_label", "difficulty"]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]


from sklearn.compose import ColumnTransformer
from sklearn.ensemble import IsolationForest
from sklearn.impute import SimpleImputer
from sklearn.metrics import ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier


from sklearn.metrics import (
    accuracy_score, average_precision_score, confusion_matrix,
    f1_score, precision_score, recall_score,
)


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "attack_precision": precision_score(y_true, y_pred, zero_division=0),
        "attack_recall": recall_score(y_true, y_pred, zero_division=0),
        "attack_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
        "alerts_per_10000": 10000 * (fp + tp) / len(y_true),
    }


OUTPUT_DIR = Path("output")
OUTPUT_DIR.mkdir(exist_ok=True)

In [ ]:

# TODO 1: load the fixed split and build the supervised feature matrices.
train_df = None
test_df = None
feature_cols = None
numeric_features = None
X_train, y_train, X_test, y_test = None, None, None, None

# TODO 2: fit the bounded, balanced Decision Tree pipeline from Lab 5.
tree_model = None
tree_predictions = None
tree_probabilities = None

1. Fit a normal-only Isolation Forest and set its alert budget

In [ ]:

# TODO 3: define anomaly_features, split normal training data, fit an
# Isolation Forest pipeline, and select the 99th validation percentile.
anomaly_features = None
normal_fit = None
normal_valid = None
anomaly_model = None
validation_scores = None
anomaly_scores = None
anomaly_threshold = None
anomaly_predictions = None

2. Compare classifier, anomaly, and hybrid policies

In [ ]:

# TODO 4: create Classifier, Anomaly, and Hybrid_OR predictions/scores.
policy_predictions = None
policy_scores = None

# TODO 5: build metrics_table and select the highest-recall policy with
# FPR <= 0.10, breaking ties by F1 then lower FPR.
metrics_table = None
selected_policy = None
selected_predictions = None
display(metrics_table)

3. Audit attack groups and create the alert evidence table

In [ ]:

# TODO 6: calculate per-group recall for every policy.
group_recall = None

# TODO 7: build alert_table, derive alert_reason, and keep the top 200
# selected alerts ordered by anomaly score and tree probability.
alert_table = None
top_alerts = None
display(top_alerts)

4. Export evidence and write the SOC recommendation

In [ ]:

# TODO 8: save a confusion matrix and anomaly-score distribution chart.

# TODO 9: write an English SOC summary covering the selected policy,
# metrics, alert volume, detector roles, weak groups, and limitations.
analyst_summary = None

# TODO 10: export the metrics, group recall, top alerts, and summary.
print(analyst_summary)

Self-check

In [ ]:

expected_files = {
    "hybrid_ids_metrics.csv", "hybrid_ids_group_recall.csv",
    "hybrid_ids_top_alerts.csv", "hybrid_ids_confusion_matrix.png",
    "hybrid_ids_score_distribution.png", "hybrid_ids_analyst_summary.md",
}
assert set(policy_predictions) == {"Classifier", "Anomaly", "Hybrid_OR"}
assert selected_policy in policy_predictions
assert len(alert_table) == len(test_df)
assert set(group_recall["attack_group"]) == {"DoS", "Probe", "R2L", "U2R"}
assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})
assert "NSL-KDD" in analyst_summary
print("✓ Lab 6 completed")

Submission checklist

1. Run the notebook from top to bottom.
2. Confirm that all six output files exist.
3. Verify that the selected policy follows the documented rule.
4. Explain how anomaly-only alerts differ from classifier-only alerts.
5. Identify the weakest attack group and one dataset limitation.