labs/Lab-5/Lab-5-Solution/03_Decision_Tree_Baseline_Solution.ipynb

Module 5 — Lab 3: Building a Decision Tree Baseline

Topics: stratified split, dummy baseline, bounded tree, leakage-safe pipeline

Solution version: Run the cells in order, compare the outputs with your work, and read the suggested answers.

Learning objectives

After completing this lab, you will be able to:

• create a reproducible stratified train-test split;
• fit missing-value imputation on training data only;
• compare a bounded Decision Tree with a dummy baseline;
• evaluate malware precision, recall, F1, false positives, and false negatives;
• explain why trees do not require feature scaling.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )


from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "malware_precision": precision_score(y_true, y_pred, zero_division=0),
        "malware_recall": recall_score(y_true, y_pred, zero_division=0),
        "malware_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
    }

In [ ]:

df = pd.read_csv(DATA_PATH, low_memory=False)
feature_cols = [column for column in df.columns if column not in {"source", "label"}]
X = df[feature_cols].apply(pd.to_numeric, errors="coerce")
y = df["label"].astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
print("Train shape:", X_train.shape, "Test shape:", X_test.shape)
print("Malware rate — train/test:", y_train.mean().round(4), y_test.mean().round(4))

1. Establish a non-learning baseline

Because malware is the majority class, a most-frequent classifier can achieve high accuracy while providing little operational value.

In [ ]:

dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_predictions = dummy.predict(X_test)
dummy_metrics = security_metrics(y_test, dummy_predictions)
pd.Series(dummy_metrics, name="dummy")

2. Train a bounded Decision Tree

The imputer is inside the pipeline, so its medians are learned from training rows only. A shallow tree is intentionally chosen for readability and to reduce overfitting.

In [ ]:

tree_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
    ("tree", DecisionTreeClassifier(
        max_depth=4,
        min_samples_leaf=10,
        random_state=42,
    )),
])
tree_pipeline.fit(X_train, y_train)
tree_predictions = tree_pipeline.predict(X_test)
tree_metrics = security_metrics(y_test, tree_predictions)

comparison = pd.DataFrame([dummy_metrics, tree_metrics], index=["Dummy", "Decision Tree"])
display(comparison.round(4))
display(pd.DataFrame(
    confusion_matrix(y_test, tree_predictions, labels=[0, 1]),
    index=["Actual benign", "Actual malware"],
    columns=["Predicted benign", "Predicted malware"],
))

Self-check

In [ ]:

assert X_train.shape[0] + X_test.shape[0] == len(df)
assert abs(y_train.mean() - y_test.mean()) < 0.01
assert set(tree_predictions).issubset({0, 1})
assert tree_pipeline.named_steps["tree"].get_depth() <= 4
assert set(comparison.index) == {"Dummy", "Decision Tree"}
assert comparison.loc["Decision Tree", "malware_f1"] >= comparison.loc["Dummy", "malware_f1"]
print("✓ Lab 3 completed")

Suggested answers

1. Why can dummy accuracy look high? Malware is the majority class, so always predicting malware is often correct but creates many false positives.
2. Why no scaler? Decision Trees compare one feature with a threshold; multiplying a feature by a constant changes the threshold but not the ordering of samples.
3. Why bound the depth? A shallow tree is easier to audit and is less likely to memorize small training regions.