labs/Lab-5/Lab-5-Practical/05_Tree_Interpretation_Overfitting_Lab.ipynb

Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting

Topics: tree rules, complexity, feature importance, train-test gap

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• compare bounded and unrestricted trees;
• quantify depth, number of leaves, and train-test performance gaps;
• visualize an explainable tree;
• export root-to-leaf logic as text rules;
• interpret feature importance cautiously.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )


from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier


from sklearn.tree import export_text, plot_tree


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "malware_precision": precision_score(y_true, y_pred, zero_division=0),
        "malware_recall": recall_score(y_true, y_pred, zero_division=0),
        "malware_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
    }

In [ ]:

# TODO 1: load data, define X/y, and make an 80/20 stratified split.
df = None
feature_cols = None
X = None
y = None
X_train, X_test, y_train, y_test = None, None, None, None

# TODO 2: implement make_tree and fit a bounded tree (depth 4, leaf 10)
# and an unrestricted tree (no maximum depth, leaf 1).
def make_tree(max_depth, min_samples_leaf):
    pass

bounded_tree = None
unrestricted_tree = None

1. Compare model complexity and generalization

In [ ]:

# TODO 3: create complexity_comparison with depth, leaves, train/test F1,
# F1 gap, test recall, and test false-positive rate for both trees.
complexity_comparison = None
display(complexity_comparison)

2. Visualize and export the bounded tree

In [ ]:

# TODO 4: obtain transformed feature names, plot the bounded tree, and
# export its rules as text with max_depth=4.
imputer = None
estimator = None
transformed_names = None
rule_text = None
print(rule_text)

3. Inspect impurity-based feature importance

In [ ]:

# TODO 5: create a descending importance Series using transformed names,
# display the top 12, and plot a horizontal bar chart.
importance = None
top_importance = None
display(top_importance)

Interpretation warning

A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation.

Self-check

In [ ]:

assert bounded_tree.named_steps["tree"].get_depth() <= 4
assert unrestricted_tree.named_steps["tree"].get_depth() >= bounded_tree.named_steps["tree"].get_depth()
assert complexity_comparison.loc["unrestricted", "leaves"] >= complexity_comparison.loc["bounded", "leaves"]
assert isinstance(rule_text, str) and "class:" in rule_text
assert np.isclose(importance.sum(), 1.0)
assert len(top_importance) <= 12
print("✓ Lab 5 completed")

Reflection questions

1. Which values in the comparison table indicate overfitting?
2. Translate two root-to-malware paths into IF ... AND ... THEN malware rules.
3. Why must feature importance not be interpreted as causal evidence?