labs/Lab-3/Lab-3-Practical/04_Classification_Metrics_Lab.ipynb

Module 3 — Lab 4: Evaluate an Intrusion Classifier

Topics: confusion matrix, accuracy, precision, recall, F1-score, error analysis

Instructions: Run the cells in order. Complete every TODO, then run the Self-check cell. Do not open the Solution notebook before attempting the tasks yourself.

Learning objectives

After completing this lab, you will be able to:

• calculate a confusion matrix and classification metrics;
• distinguish false positives from false negatives;
• explain why attack recall matters in intrusion detection;
• inspect misclassified network flows.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

DATA_PATH = Path("data/cicids2017_binary_sample.csv")
RANDOM_STATE = 42

import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    ConfusionMatrixDisplay, accuracy_score, classification_report,
    confusion_matrix, precision_score, recall_score, f1_score
)

1. Train the baseline pipeline

In [ ]:

df = pd.read_csv(DATA_PATH).drop_duplicates()
df.columns = df.columns.str.strip()
df = df.replace([np.inf, -np.inf], np.nan)

X = df.drop(columns="Label").select_dtypes(include="number")
y = df["Label"].astype(str).str.strip().str.upper().ne("BENIGN").astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y
)

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("model", RandomForestClassifier(
        n_estimators=100, max_depth=12,
        random_state=RANDOM_STATE, n_jobs=-1
    )),
])

# TODO 1: fit the pipeline and create y_pred
y_pred = None

2. Calculate the confusion matrix

In [ ]:

# TODO 2: calculate the confusion matrix and unpack tn, fp, fn, tp
cm = None
tn, fp, fn, tp = None, None, None, None

print("Confusion matrix:\n", cm)
print(f"TN={tn}, FP={fp}, FN={fn}, TP={tp}")

# TODO 3: display the confusion matrix with class names BENIGN and ATTACK

3. Calculate classification metrics

In [ ]:

# TODO 4: calculate accuracy, attack precision, attack recall, and attack F1
accuracy = None
attack_precision = None
attack_recall = None
attack_f1 = None

metrics = pd.Series({
    "accuracy": accuracy,
    "attack_precision": attack_precision,
    "attack_recall": attack_recall,
    "attack_f1": attack_f1,
})
print(metrics.round(3))

# TODO 5: print classification_report with BENIGN and ATTACK target names

4. Inspect false negatives

In [ ]:

results = X_test.copy()
results["actual"] = y_test
results["predicted"] = y_pred

# TODO 6: retain rows where actual is ATTACK and predicted is BENIGN
false_negatives = None

print("False negatives:", None)
false_negatives.head()

Self-check

In [ ]:

assert cm.shape == (2, 2)
assert tn + fp + fn + tp == len(y_test)
assert 0 <= accuracy <= 1
assert 0 <= attack_precision <= 1
assert 0 <= attack_recall <= 1
assert 0 <= attack_f1 <= 1
assert len(false_negatives) == fn
print("✓ Lab 4 completed")

Analyst questions

1. Which confusion-matrix cell represents a missed attack?
2. Which metric should receive more attention if missed attacks are very costly?
3. Why can improving attack recall increase the number of false alerts?