{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 5 — Lab 4: Security Metrics and Error Costs\n",
    "\n",
    " \n",
    "**Topics:** false negatives, class weights, validation threshold, test discipline\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- quantify false-negative and false-positive costs;\n",
    "- compare class-weight policies on validation data;\n",
    "- select a probability threshold without inspecting the test set;\n",
    "- evaluate the chosen policy once on held-out test data;\n",
    "- describe the operational trade-off between missed malware and analyst workload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/malware_features.csv\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
    "        \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
    "    )\n",
    "\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "\n",
    "def security_metrics(y_true, y_pred):\n",
    "    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
    "    return {\n",
    "        \"accuracy\": accuracy_score(y_true, y_pred),\n",
    "        \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
    "        \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
    "        \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
    "        \"false_positives\": int(fp),\n",
    "        \"false_negatives\": int(fn),\n",
    "        \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load X and y, create a 20% stratified test set, then split the\n",
    "# remaining data into 75% train and 25% validation using random_state=42.\n",
    "df = None\n",
    "feature_cols = None\n",
    "X = None\n",
    "y = None\n",
    "X_train_full, X_test, y_train_full, y_test = None, None, None, None\n",
    "X_train, X_valid, y_train, y_valid = None, None, None, None\n",
    "print(\"Train/validation/test:\", None, None, None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Compare class-weight policies on validation data\n",
    "\n",
    "Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "weight_policies = {\n",
    "    \"unweighted\": None,\n",
    "    \"malware_x2\": {0: 1, 1: 2},\n",
    "    \"malware_x4\": {0: 1, 1: 4},\n",
    "}\n",
    "\n",
    "# TODO 2: fit one depth-4 tree pipeline per weight policy and collect\n",
    "# validation security_metrics in validation_results.\n",
    "validation_results = None\n",
    "fitted_candidates = None\n",
    "display(validation_results)\n",
    "\n",
    "# TODO 3: select highest recall, breaking ties by lower FPR then higher F1.\n",
    "selected_policy = None\n",
    "selected_model = None\n",
    "print(\"Selected class-weight policy:\", selected_policy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Select a validation threshold\n",
    "\n",
    "Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 4: obtain validation malware probabilities and evaluate thresholds\n",
    "# from 0.20 through 0.80 in steps of 0.05.\n",
    "validation_probabilities = None\n",
    "threshold_results = None\n",
    "\n",
    "# TODO 5: among thresholds with recall >= 0.98, select the lowest FPR.\n",
    "# If none qualify, use a threshold with maximum recall.\n",
    "selected_threshold = None\n",
    "display(threshold_results)\n",
    "print(\"Selected threshold:\", selected_threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## 3. Refit on all training data and evaluate once on the test set\n",
    "\n",
    "The test set is used only after all policy decisions are fixed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-010",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 6: refit the selected policy on X_train_full/y_train_full.\n",
    "final_model = None\n",
    "\n",
    "# TODO 7: compare threshold 0.50 and selected_threshold on the test set.\n",
    "test_probabilities = None\n",
    "default_predictions = None\n",
    "tuned_predictions = None\n",
    "test_results = None\n",
    "display(test_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-012",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(validation_results.index) == set(weight_policies)\n",
    "assert 0.20 <= selected_threshold <= 0.80\n",
    "assert len(test_probabilities) == len(y_test)\n",
    "assert set(tuned_predictions).issubset({0, 1})\n",
    "assert test_results.shape[0] == 2\n",
    "assert final_model.named_steps[\"tree\"].get_depth() <= 4\n",
    "print(\"✓ Lab 4 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-013",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why must model and threshold choices use validation data rather than test data?\n",
    "2. What operational harm can false negatives and false positives cause?\n",
    "3. Why should the selected threshold be revalidated after deployment conditions change?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}