{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
    "\n",
    "\n",
    "**Topics:** feature families, distributions, effect size, responsible EDA\n",
    "\n",
    "> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- summarize numeric malware features by class;\n",
    "- identify interpretable feature-name families;\n",
    "- compare benign and malware distributions with robust statistics;\n",
    "- distinguish exploratory association from causal evidence;\n",
    "- avoid using the held-out test set for model-driven feature selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/malware_features.csv\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
    "        \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-004",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, low_memory=False)\n",
    "feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
    "X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
    "y = df[\"label\"].astype(int)\n",
    "print(\"Rows:\", len(df), \"Features:\", len(feature_cols))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-005",
   "metadata": {},
   "source": [
    "## 1. Inspect feature-name families\n",
    "\n",
    "Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-006",
   "metadata": {},
   "outputs": [],
   "source": [
    "family_tokens = {\n",
    "    \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
    "    \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
    "    \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
    "    \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
    "}\n",
    "\n",
    "family_examples = {}\n",
    "for family, tokens in family_tokens.items():\n",
    "    matches = [name for name in feature_cols if any(token in name.lower() for token in tokens)]\n",
    "    family_examples[family] = matches[:10]\n",
    "family_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-007",
   "metadata": {},
   "source": [
    "## 2. Build a robust class-comparison profile\n",
    "\n",
    "Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-008",
   "metadata": {},
   "outputs": [],
   "source": [
    "benign = X.loc[y == 0]\n",
    "malware = X.loc[y == 1]\n",
    "q75 = X.quantile(0.75)\n",
    "q25 = X.quantile(0.25)\n",
    "iqr = (q75 - q25).replace(0, np.nan)\n",
    "\n",
    "feature_profile = pd.DataFrame({\n",
    "    \"missing_rate\": X.isna().mean(),\n",
    "    \"unique_values\": X.nunique(dropna=True),\n",
    "    \"benign_median\": benign.median(),\n",
    "    \"malware_median\": malware.median(),\n",
    "})\n",
    "feature_profile[\"standardized_median_gap\"] = (\n",
    "    (feature_profile[\"malware_median\"] - feature_profile[\"benign_median\"]).abs() / iqr\n",
    ")\n",
    "feature_profile = feature_profile.replace([np.inf, -np.inf], np.nan)\n",
    "top_features = feature_profile[\"standardized_median_gap\"].nlargest(8).index.tolist()\n",
    "display(feature_profile.loc[top_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-009",
   "metadata": {},
   "source": [
    "## 3. Visualize class medians for the strongest exploratory differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-010",
   "metadata": {},
   "outputs": [],
   "source": [
    "median_plot = feature_profile.loc[top_features, [\"benign_median\", \"malware_median\"]]\n",
    "ax = median_plot.plot(kind=\"barh\", figsize=(10, 6))\n",
    "ax.set_title(\"Class medians for exploratory high-gap features\")\n",
    "ax.set_xlabel(\"Median feature value\")\n",
    "ax.set_ylabel(\"Feature\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## Interpretation warning\n",
    "\n",
    "A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-012",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-013",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(X.columns) == set(feature_cols)\n",
    "assert set(y.unique()) == {0, 1}\n",
    "assert len(feature_profile) == len(feature_cols)\n",
    "assert len(top_features) == 8\n",
    "assert set(top_features).issubset(feature_cols)\n",
    "assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
    "print(\"✓ Lab 2 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-014",
   "metadata": {},
   "source": [
    "## Suggested answers\n",
    "\n",
    "1. **Why use medians and IQR?** They are less sensitive to extreme values and skew than means and standard deviations.\n",
    "2. **Can the ranking prove causation?** No. It identifies associations in this particular collection.\n",
    "3. **Why not choose final features from the full table?** Doing so lets test-set patterns influence model design and makes the final score optimistic."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}