labs/Lab-5/Lab-5-Solution/05_Tree_Interpretation_Overfitting_Solution.ipynb

Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting

Topics: tree rules, complexity, feature importance, train-test gap

Solution version: Run the cells in order, compare the outputs with your work, and read the suggested answers.

Learning objectives

After completing this lab, you will be able to:

• compare bounded and unrestricted trees;
• quantify depth, number of leaves, and train-test performance gaps;
• visualize an explainable tree;
• export root-to-leaf logic as text rules;
• interpret feature importance cautiously.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )


from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier


from sklearn.tree import export_text, plot_tree


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "malware_precision": precision_score(y_true, y_pred, zero_division=0),
        "malware_recall": recall_score(y_true, y_pred, zero_division=0),
        "malware_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
    }

In [ ]:

df = pd.read_csv(DATA_PATH, low_memory=False)
feature_cols = [column for column in df.columns if column not in {"source", "label"}]
X = df[feature_cols].apply(pd.to_numeric, errors="coerce")
y = df["label"].astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

def make_tree(max_depth, min_samples_leaf):
    return Pipeline([
        ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
        ("tree", DecisionTreeClassifier(
            max_depth=max_depth, min_samples_leaf=min_samples_leaf, random_state=42
        )),
    ])

bounded_tree = make_tree(max_depth=4, min_samples_leaf=10)
unrestricted_tree = make_tree(max_depth=None, min_samples_leaf=1)
bounded_tree.fit(X_train, y_train)
unrestricted_tree.fit(X_train, y_train)

1. Compare model complexity and generalization

In [ ]:

comparison_rows = []
for name, model in {"bounded": bounded_tree, "unrestricted": unrestricted_tree}.items():
    estimator = model.named_steps["tree"]
    train_metrics = security_metrics(y_train, model.predict(X_train))
    test_metrics = security_metrics(y_test, model.predict(X_test))
    comparison_rows.append({
        "model": name,
        "depth": estimator.get_depth(),
        "leaves": estimator.get_n_leaves(),
        "train_f1": train_metrics["malware_f1"],
        "test_f1": test_metrics["malware_f1"],
        "f1_gap": train_metrics["malware_f1"] - test_metrics["malware_f1"],
        "test_recall": test_metrics["malware_recall"],
        "test_fpr": test_metrics["false_positive_rate"],
    })
complexity_comparison = pd.DataFrame(comparison_rows).set_index("model")
display(complexity_comparison.round(4))

2. Visualize and export the bounded tree

In [ ]:

imputer = bounded_tree.named_steps["imputer"]
estimator = bounded_tree.named_steps["tree"]
transformed_names = imputer.get_feature_names_out(feature_cols)

plt.figure(figsize=(24, 12))
plot_tree(
    estimator,
    feature_names=transformed_names,
    class_names=["benign", "malware"],
    filled=True,
    rounded=True,
    proportion=True,
    fontsize=7,
)
plt.title("Bounded malware Decision Tree")
plt.tight_layout()
plt.show()

rule_text = export_text(estimator, feature_names=list(transformed_names), max_depth=4)
print(rule_text)

3. Inspect impurity-based feature importance

In [ ]:

importance = pd.Series(
    estimator.feature_importances_, index=transformed_names, name="importance"
).sort_values(ascending=False)
top_importance = importance.head(12)
display(top_importance.to_frame())

ax = top_importance.sort_values().plot(kind="barh", figsize=(9, 6), color="#4C78A8")
ax.set_title("Top bounded-tree feature importances")
ax.set_xlabel("Mean decrease in impurity")
plt.tight_layout()
plt.show()

Interpretation warning

A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation.

Self-check

In [ ]:

assert bounded_tree.named_steps["tree"].get_depth() <= 4
assert unrestricted_tree.named_steps["tree"].get_depth() >= bounded_tree.named_steps["tree"].get_depth()
assert complexity_comparison.loc["unrestricted", "leaves"] >= complexity_comparison.loc["bounded", "leaves"]
assert isinstance(rule_text, str) and "class:" in rule_text
assert np.isclose(importance.sum(), 1.0)
assert len(top_importance) <= 12
print("✓ Lab 5 completed")

Suggested answers

1. What indicates overfitting? The unrestricted tree has more depth/leaves and typically a larger train-test performance gap.
2. How should a rule be written? Combine each split along one root-to-leaf path with AND, then attach the leaf prediction and sample support.
3. Why is importance not causality? It measures impurity reduction inside one fitted model and can reflect correlation or collection artifacts.