276 lines
8.3 KiB
Plaintext
276 lines
8.3 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-001",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow\n",
|
|
"\n",
|
|
"\n",
|
|
"**Topics:** end-to-end workflow, model comparison, error evidence, analyst recommendation\n",
|
|
"\n",
|
|
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-002",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Mission\n",
|
|
"\n",
|
|
"Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.\n",
|
|
"\n",
|
|
"**Required outputs**\n",
|
|
"\n",
|
|
"- `output/capstone_metrics.csv`\n",
|
|
"- `output/capstone_errors.csv`\n",
|
|
"- `output/capstone_tree_rules.txt`\n",
|
|
"- `output/capstone_confusion_matrix.png`\n",
|
|
"- `output/capstone_analyst_summary.md`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-003",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pathlib import Path\n",
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
|
|
"\n",
|
|
"if not DATA_PATH.exists():\n",
|
|
" raise FileNotFoundError(\n",
|
|
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
|
|
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
|
|
" )\n",
|
|
"\n",
|
|
"\n",
|
|
"from sklearn.dummy import DummyClassifier\n",
|
|
"from sklearn.impute import SimpleImputer\n",
|
|
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.pipeline import Pipeline\n",
|
|
"from sklearn.tree import DecisionTreeClassifier\n",
|
|
"\n",
|
|
"\n",
|
|
"from sklearn.ensemble import RandomForestClassifier\n",
|
|
"from sklearn.metrics import ConfusionMatrixDisplay\n",
|
|
"from sklearn.tree import export_text\n",
|
|
"\n",
|
|
"\n",
|
|
"def security_metrics(y_true, y_pred):\n",
|
|
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
|
|
" return {\n",
|
|
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
|
|
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
|
|
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
|
|
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
|
|
" \"false_positives\": int(fp),\n",
|
|
" \"false_negatives\": int(fn),\n",
|
|
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
|
|
" }\n",
|
|
"\n",
|
|
"\n",
|
|
"OUTPUT_DIR = Path(\"output\")\n",
|
|
"OUTPUT_DIR.mkdir(exist_ok=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-004",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 1: load the dataset, exclude source/label from X, and create an\n",
|
|
"# 80/20 stratified split that also retains source_test for error auditing.\n",
|
|
"df = None\n",
|
|
"feature_cols = None\n",
|
|
"X = None\n",
|
|
"y = None\n",
|
|
"X_train, X_test, y_train, y_test, source_train, source_test = (None,) * 6\n",
|
|
"print(\"Train/test rows:\", None, None)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-005",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 1. Build and compare candidate models"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-006",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 2: define Dummy, ExplainableTree, DeepTree, and RandomForest.\n",
|
|
"# Keep ExplainableTree at max_depth=4 and min_samples_leaf=10.\n",
|
|
"models = None\n",
|
|
"\n",
|
|
"# TODO 3: fit every model, save predictions/probabilities, and build a\n",
|
|
"# metrics table with security_metrics.\n",
|
|
"metric_rows = None\n",
|
|
"predictions = None\n",
|
|
"probabilities = None\n",
|
|
"metrics_table = None\n",
|
|
"display(metrics_table)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-007",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 2. Select a model with an explicit decision rule\n",
|
|
"\n",
|
|
"Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-008",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 4: implement the documented selection rule.\n",
|
|
"eligible_metrics = None\n",
|
|
"best_recall = None\n",
|
|
"best_fpr = None\n",
|
|
"explainable_ok = None\n",
|
|
"selected_name = None\n",
|
|
"selected_model = None\n",
|
|
"selected_predictions = None\n",
|
|
"print(\"Selected model:\", selected_name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-009",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3. Create error evidence and a confusion matrix"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-010",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 5: create error_table with source, true label, prediction,\n",
|
|
"# malware probability, and error_type. Keep mistakes in errors_only.\n",
|
|
"error_table = None\n",
|
|
"errors_only = None\n",
|
|
"display(errors_only[\"error_type\"].value_counts())\n",
|
|
"\n",
|
|
"# TODO 6: plot and save the selected model's confusion matrix."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-011",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 4. Export rules and write the analyst recommendation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-012",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# TODO 7: export the ExplainableTree rules even if another model was selected.\n",
|
|
"tree_rules = None\n",
|
|
"\n",
|
|
"# TODO 8: write an English analyst summary covering the recommendation,\n",
|
|
"# metrics, FP/FN costs, data limitations, and deployment safeguards.\n",
|
|
"analyst_summary = None\n",
|
|
"\n",
|
|
"# TODO 9: save the metrics, errors, rules, and analyst summary.\n",
|
|
"print(analyst_summary)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-013",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Self-check"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cell-014",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"expected_files = {\n",
|
|
" \"capstone_metrics.csv\",\n",
|
|
" \"capstone_errors.csv\",\n",
|
|
" \"capstone_tree_rules.txt\",\n",
|
|
" \"capstone_confusion_matrix.png\",\n",
|
|
" \"capstone_analyst_summary.md\",\n",
|
|
"}\n",
|
|
"assert set(models) == {\"Dummy\", \"ExplainableTree\", \"DeepTree\", \"RandomForest\"}\n",
|
|
"assert selected_name in set(models) - {\"Dummy\"}\n",
|
|
"assert len(error_table) == len(y_test)\n",
|
|
"assert set(errors_only[\"error_type\"].unique()).issubset({\"false_negative\", \"false_positive\"})\n",
|
|
"assert isinstance(tree_rules, str) and \"class:\" in tree_rules\n",
|
|
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
|
|
"print(\"✓ Lab 6 completed\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cell-015",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Submission checklist\n",
|
|
"\n",
|
|
"1. Run the notebook from top to bottom.\n",
|
|
"2. Confirm that all five output files exist.\n",
|
|
"3. Verify that the recommendation is supported by the metrics table.\n",
|
|
"4. Explain at least one false negative and one false positive operationally.\n",
|
|
"5. State why this random-split result is not a production guarantee."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|