labs/Lab-5/Lab-5-Practical/04_Security_Metrics_Error_Costs_Lab.ipynb

Module 5 — Lab 4: Security Metrics and Error Costs

Topics: false negatives, class weights, validation threshold, test discipline

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• quantify false-negative and false-positive costs;
• compare class-weight policies on validation data;
• select a probability threshold without inspecting the test set;
• evaluate the chosen policy once on held-out test data;
• describe the operational trade-off between missed malware and analyst workload.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )


from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "malware_precision": precision_score(y_true, y_pred, zero_division=0),
        "malware_recall": recall_score(y_true, y_pred, zero_division=0),
        "malware_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
    }

In [ ]:

# TODO 1: load X and y, create a 20% stratified test set, then split the
# remaining data into 75% train and 25% validation using random_state=42.
df = None
feature_cols = None
X = None
y = None
X_train_full, X_test, y_train_full, y_test = None, None, None, None
X_train, X_valid, y_train, y_valid = None, None, None, None
print("Train/validation/test:", None, None, None)

1. Compare class-weight policies on validation data

Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity.

In [ ]:

weight_policies = {
    "unweighted": None,
    "malware_x2": {0: 1, 1: 2},
    "malware_x4": {0: 1, 1: 4},
}

# TODO 2: fit one depth-4 tree pipeline per weight policy and collect
# validation security_metrics in validation_results.
validation_results = None
fitted_candidates = None
display(validation_results)

# TODO 3: select highest recall, breaking ties by lower FPR then higher F1.
selected_policy = None
selected_model = None
print("Selected class-weight policy:", selected_policy)

2. Select a validation threshold

Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate.

In [ ]:

# TODO 4: obtain validation malware probabilities and evaluate thresholds
# from 0.20 through 0.80 in steps of 0.05.
validation_probabilities = None
threshold_results = None

# TODO 5: among thresholds with recall >= 0.98, select the lowest FPR.
# If none qualify, use a threshold with maximum recall.
selected_threshold = None
display(threshold_results)
print("Selected threshold:", selected_threshold)

3. Refit on all training data and evaluate once on the test set

The test set is used only after all policy decisions are fixed.

In [ ]:

# TODO 6: refit the selected policy on X_train_full/y_train_full.
final_model = None

# TODO 7: compare threshold 0.50 and selected_threshold on the test set.
test_probabilities = None
default_predictions = None
tuned_predictions = None
test_results = None
display(test_results)

Self-check

In [ ]:

assert set(validation_results.index) == set(weight_policies)
assert 0.20 <= selected_threshold <= 0.80
assert len(test_probabilities) == len(y_test)
assert set(tuned_predictions).issubset({0, 1})
assert test_results.shape[0] == 2
assert final_model.named_steps["tree"].get_depth() <= 4
print("✓ Lab 4 completed")

Reflection questions

1. Why must model and threshold choices use validation data rather than test data?
2. What operational harm can false negatives and false positives cause?
3. Why should the selected threshold be revalidated after deployment conditions change?