labs/Lab-3/Lab-3-Practical/05_Class_Imbalance_Experiment_Lab.ipynb

Module 3 — Lab 5: A Controlled Class-Imbalance Experiment

Topics: class imbalance, class weights, controlled comparison, metric trade-offs

Instructions: Run the cells in order. Complete every TODO, then run the Self-check cell. Do not open the Solution notebook before attempting the tasks yourself.

Learning objectives

After completing this lab, you will be able to:

• measure the class imbalance in a security dataset;
• compare two runs of the same model on the same split;
• use class_weight="balanced" without changing the rest of the workflow;
• select a model configuration using attack precision, recall, and F1.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

DATA_PATH = Path("data/synthetic_security_flows.csv")
RANDOM_STATE = 42

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix

1. Prepare one fixed experiment

In [ ]:

df = pd.read_csv(DATA_PATH).drop_duplicates()
df.columns = df.columns.str.strip()
df = df.replace([np.inf, -np.inf], np.nan)

X = df.drop(columns="Label").select_dtypes(include="number")
y = df["Label"].astype(str).str.strip().str.upper().ne("BENIGN").astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y
)

# TODO 1: calculate benign_count, attack_count, and imbalance_ratio
benign_count = None
attack_count = None
imbalance_ratio = None

print(f"BENIGN={benign_count}, ATTACK={attack_count}")
print(f"BENIGN-to-ATTACK ratio: {imbalance_ratio:.2f}:1")

2. Define a reusable training function

In [ ]:

def train_and_measure(class_weight):
    # TODO 2: create the imputer + Random Forest pipeline
    model = None

    # TODO 3: fit, predict, and return a metrics dictionary
    return None

3. Run the controlled comparison

In [ ]:

# Keep the data split and all hyperparameters fixed.
# TODO 4: run the baseline with class_weight=None
baseline_metrics = None

# TODO 5: run the weighted model with class_weight="balanced"
balanced_metrics = None

# TODO 6: create comparison_df with Baseline and Balanced rows
comparison_df = None

comparison_df.round(3)

4. Make an evidence-based recommendation

In [ ]:

# TODO 7: select the configuration with the higher attack F1 score
recommended_configuration = None

print("Recommended configuration:", recommended_configuration)
print(comparison_df.loc[recommended_configuration].round(3))

Self-check

In [ ]:

assert benign_count > attack_count
assert imbalance_ratio > 1
assert list(comparison_df.index) == ["Baseline", "Balanced"]
assert {"precision", "recall", "f1", "false_positives", "false_negatives"}.issubset(comparison_df.columns)
assert comparison_df[["precision", "recall", "f1"]].apply(lambda col: col.between(0, 1).all()).all()
assert recommended_configuration in {"Baseline", "Balanced"}
print("✓ Lab 5 completed")

Analyst questions

1. Did class weighting improve attack recall in this run?
2. What happened to false positives and false negatives?
3. Would you choose the same configuration if analyst time were extremely limited?
4. Why is changing only one variable important in a controlled experiment?