{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow\n",
    "\n",
    "\n",
    "**Topics:** end-to-end workflow, model comparison, error evidence, analyst recommendation\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Mission\n",
    "\n",
    "Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.\n",
    "\n",
    "**Required outputs**\n",
    "\n",
    "- `output/capstone_metrics.csv`\n",
    "- `output/capstone_errors.csv`\n",
    "- `output/capstone_tree_rules.txt`\n",
    "- `output/capstone_confusion_matrix.png`\n",
    "- `output/capstone_analyst_summary.md`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/malware_features.csv\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
    "        \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
    "    )\n",
    "\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import ConfusionMatrixDisplay\n",
    "from sklearn.tree import export_text\n",
    "\n",
    "\n",
    "def security_metrics(y_true, y_pred):\n",
    "    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
    "    return {\n",
    "        \"accuracy\": accuracy_score(y_true, y_pred),\n",
    "        \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
    "        \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
    "        \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
    "        \"false_positives\": int(fp),\n",
    "        \"false_negatives\": int(fn),\n",
    "        \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
    "    }\n",
    "\n",
    "\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load the dataset, exclude source/label from X, and create an\n",
    "# 80/20 stratified split that also retains source_test for error auditing.\n",
    "df = None\n",
    "feature_cols = None\n",
    "X = None\n",
    "y = None\n",
    "X_train, X_test, y_train, y_test, source_train, source_test = (None,) * 6\n",
    "print(\"Train/test rows:\", None, None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Build and compare candidate models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 2: define Dummy, ExplainableTree, DeepTree, and RandomForest.\n",
    "# Keep ExplainableTree at max_depth=4 and min_samples_leaf=10.\n",
    "models = None\n",
    "\n",
    "# TODO 3: fit every model, save predictions/probabilities, and build a\n",
    "# metrics table with security_metrics.\n",
    "metric_rows = None\n",
    "predictions = None\n",
    "probabilities = None\n",
    "metrics_table = None\n",
    "display(metrics_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Select a model with an explicit decision rule\n",
    "\n",
    "Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 4: implement the documented selection rule.\n",
    "eligible_metrics = None\n",
    "best_recall = None\n",
    "best_fpr = None\n",
    "explainable_ok = None\n",
    "selected_name = None\n",
    "selected_model = None\n",
    "selected_predictions = None\n",
    "print(\"Selected model:\", selected_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## 3. Create error evidence and a confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-010",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 5: create error_table with source, true label, prediction,\n",
    "# malware probability, and error_type. Keep mistakes in errors_only.\n",
    "error_table = None\n",
    "errors_only = None\n",
    "display(errors_only[\"error_type\"].value_counts())\n",
    "\n",
    "# TODO 6: plot and save the selected model's confusion matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## 4. Export rules and write the analyst recommendation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-012",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: export the ExplainableTree rules even if another model was selected.\n",
    "tree_rules = None\n",
    "\n",
    "# TODO 8: write an English analyst summary covering the recommendation,\n",
    "# metrics, FP/FN costs, data limitations, and deployment safeguards.\n",
    "analyst_summary = None\n",
    "\n",
    "# TODO 9: save the metrics, errors, rules, and analyst summary.\n",
    "print(analyst_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-013",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-014",
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_files = {\n",
    "    \"capstone_metrics.csv\",\n",
    "    \"capstone_errors.csv\",\n",
    "    \"capstone_tree_rules.txt\",\n",
    "    \"capstone_confusion_matrix.png\",\n",
    "    \"capstone_analyst_summary.md\",\n",
    "}\n",
    "assert set(models) == {\"Dummy\", \"ExplainableTree\", \"DeepTree\", \"RandomForest\"}\n",
    "assert selected_name in set(models) - {\"Dummy\"}\n",
    "assert len(error_table) == len(y_test)\n",
    "assert set(errors_only[\"error_type\"].unique()).issubset({\"false_negative\", \"false_positive\"})\n",
    "assert isinstance(tree_rules, str) and \"class:\" in tree_rules\n",
    "assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
    "print(\"✓ Lab 6 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-015",
   "metadata": {},
   "source": [
    "## Submission checklist\n",
    "\n",
    "1. Run the notebook from top to bottom.\n",
    "2. Confirm that all five output files exist.\n",
    "3. Verify that the recommendation is supported by the metrics table.\n",
    "4. Explain at least one false negative and one false positive operationally.\n",
    "5. State why this random-split result is not a production guarantee."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}