labs/Lab-5/Lab-5-Practical/06_Capstone_Malware_Detection_Lab.ipynb

Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow

Topics: end-to-end workflow, model comparison, error evidence, analyst recommendation

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Mission

Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.

Required outputs

• output/capstone_metrics.csv
• output/capstone_errors.csv
• output/capstone_tree_rules.txt
• output/capstone_confusion_matrix.png
• output/capstone_analyst_summary.md

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )


from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier


from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import ConfusionMatrixDisplay
from sklearn.tree import export_text


def security_metrics(y_true, y_pred):
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "malware_precision": precision_score(y_true, y_pred, zero_division=0),
        "malware_recall": recall_score(y_true, y_pred, zero_division=0),
        "malware_f1": f1_score(y_true, y_pred, zero_division=0),
        "false_positives": int(fp),
        "false_negatives": int(fn),
        "false_positive_rate": fp / (fp + tn) if fp + tn else 0.0,
    }


OUTPUT_DIR = Path("output")
OUTPUT_DIR.mkdir(exist_ok=True)

In [ ]:

# TODO 1: load the dataset, exclude source/label from X, and create an
# 80/20 stratified split that also retains source_test for error auditing.
df = None
feature_cols = None
X = None
y = None
X_train, X_test, y_train, y_test, source_train, source_test = (None,) * 6
print("Train/test rows:", None, None)

1. Build and compare candidate models

In [ ]:

# TODO 2: define Dummy, ExplainableTree, DeepTree, and RandomForest.
# Keep ExplainableTree at max_depth=4 and min_samples_leaf=10.
models = None

# TODO 3: fit every model, save predictions/probabilities, and build a
# metrics table with security_metrics.
metric_rows = None
predictions = None
probabilities = None
metrics_table = None
display(metrics_table)

2. Select a model with an explicit decision rule

Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1.

In [ ]:

# TODO 4: implement the documented selection rule.
eligible_metrics = None
best_recall = None
best_fpr = None
explainable_ok = None
selected_name = None
selected_model = None
selected_predictions = None
print("Selected model:", selected_name)

3. Create error evidence and a confusion matrix

In [ ]:

# TODO 5: create error_table with source, true label, prediction,
# malware probability, and error_type. Keep mistakes in errors_only.
error_table = None
errors_only = None
display(errors_only["error_type"].value_counts())

# TODO 6: plot and save the selected model's confusion matrix.

4. Export rules and write the analyst recommendation

In [ ]:

# TODO 7: export the ExplainableTree rules even if another model was selected.
tree_rules = None

# TODO 8: write an English analyst summary covering the recommendation,
# metrics, FP/FN costs, data limitations, and deployment safeguards.
analyst_summary = None

# TODO 9: save the metrics, errors, rules, and analyst summary.
print(analyst_summary)

Self-check

In [ ]:

expected_files = {
    "capstone_metrics.csv",
    "capstone_errors.csv",
    "capstone_tree_rules.txt",
    "capstone_confusion_matrix.png",
    "capstone_analyst_summary.md",
}
assert set(models) == {"Dummy", "ExplainableTree", "DeepTree", "RandomForest"}
assert selected_name in set(models) - {"Dummy"}
assert len(error_table) == len(y_test)
assert set(errors_only["error_type"].unique()).issubset({"false_negative", "false_positive"})
assert isinstance(tree_rules, str) and "class:" in tree_rules
assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})
print("✓ Lab 6 completed")

Submission checklist

1. Run the notebook from top to bottom.
2. Confirm that all five output files exist.
3. Verify that the recommendation is supported by the metrics table.
4. Explain at least one false negative and one false positive operationally.
5. State why this random-split result is not a production guarantee.