{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
    "\n",
    " \n",
    "**Topics:** feature families, distributions, effect size, responsible EDA\n",
    "\n",
    "> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- summarize numeric malware features by class;\n",
    "- identify interpretable feature-name families;\n",
    "- compare benign and malware distributions with robust statistics;\n",
    "- distinguish exploratory association from causal evidence;\n",
    "- avoid using the held-out test set for model-driven feature selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/malware_features.csv\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
    "        \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 1: load the prepared dataset and define X, y, and feature_cols.\n",
    "df = None\n",
    "feature_cols = None\n",
    "X = None\n",
    "y = None\n",
    "print(\"Rows:\", None, \"Features:\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Inspect feature-name families\n",
    "\n",
    "Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "family_tokens = {\n",
    "    \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
    "    \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
    "    \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
    "    \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
    "}\n",
    "\n",
    "# TODO 2: collect up to ten feature-name examples for each family.\n",
    "family_examples = None\n",
    "family_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Build a robust class-comparison profile\n",
    "\n",
    "Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "benign = X.loc[y == 0]\n",
    "malware = X.loc[y == 1]\n",
    "\n",
    "# TODO 3: calculate the IQR for every feature and replace zero IQR values with NaN.\n",
    "iqr = None\n",
    "\n",
    "# TODO 4: build feature_profile with missing rate, unique count, and class medians.\n",
    "feature_profile = None\n",
    "\n",
    "# TODO 5: calculate the absolute median gap divided by IQR, then select the top eight names.\n",
    "top_features = None\n",
    "display(feature_profile.loc[top_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## 3. Visualize class medians for the strongest exploratory differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-010",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 6: create a horizontal grouped bar chart of benign and malware medians.\n",
    "median_plot = None\n",
    "ax = None\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## Interpretation warning\n",
    "\n",
    "A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(X.columns) == set(feature_cols)\n",
    "assert set(y.unique()) == {0, 1}\n",
    "assert len(feature_profile) == len(feature_cols)\n",
    "assert len(top_features) == 8\n",
    "assert set(top_features).issubset(feature_cols)\n",
    "assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
    "print(\"✓ Lab 2 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-014",
   "metadata": {},
   "source": [
    "## Reflection questions\n",
    "\n",
    "1. Why are medians and IQR useful for security features?\n",
    "2. Can a large standardized median gap prove that a feature causes malware behavior?\n",
    "3. Why should this full-dataset ranking not be used directly to optimize the final model?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}