{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-001",
   "metadata": {},
   "source": [
    "# Module 5 — Lab 1: Auditing a Malware Feature Dataset\n",
    "\n",
    "\n",
    "**Topics:** provenance, schema, imbalance, missing values, leakage\n",
    "\n",
    "> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-002",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- load a prepared table of static and dynamic malware features safely;\n",
    "- distinguish identifiers, provenance fields, features, and the binary label;\n",
    "- audit class imbalance, source composition, missing values, infinite values, and duplicates;\n",
    "- identify columns that must not enter a predictive model;\n",
    "- document dataset limitations before modeling.\n",
    "\n",
    "> **Safety boundary:** This practical uses numeric feature tables only. Do not download, open, or execute malware samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-003",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "DATA_PATH = Path(\"data/malware_features.csv\")\n",
    "\n",
    "if not DATA_PATH.exists():\n",
    "    raise FileNotFoundError(\n",
    "        \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
    "        \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-004",
   "metadata": {},
   "source": [
    "## 1. Load the prepared table\n",
    "\n",
    "The preparation utility adds two non-feature columns: `source` records provenance and `label` uses `0 = benign`, `1 = malware`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-005",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH, low_memory=False)\n",
    "required_columns = {\"source\", \"label\"}\n",
    "if not required_columns.issubset(df.columns):\n",
    "    raise ValueError(f\"Missing required columns: {required_columns - set(df.columns)}\")\n",
    "\n",
    "feature_cols = [column for column in df.columns if column not in required_columns]\n",
    "print(\"Dataset shape:\", df.shape)\n",
    "print(\"Number of candidate features:\", len(feature_cols))\n",
    "display(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-006",
   "metadata": {},
   "source": [
    "## 2. Audit labels and provenance\n",
    "\n",
    "`source` is useful for detecting collection bias, but it must never be used as a predictor. In this dataset, collection source and class are closely related."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-007",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_counts = df[\"label\"].value_counts().sort_index()\n",
    "class_proportions = df[\"label\"].value_counts(normalize=True).sort_index()\n",
    "source_by_label = pd.crosstab(df[\"source\"], df[\"label\"], margins=True)\n",
    "\n",
    "print(\"Class counts (0 = benign, 1 = malware):\")\n",
    "display(pd.DataFrame({\"count\": class_counts, \"proportion\": class_proportions}))\n",
    "print(\"Source by label:\")\n",
    "display(source_by_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-008",
   "metadata": {},
   "source": [
    "## 3. Audit data quality and leakage risks\n",
    "\n",
    "Missing values will be imputed inside a training pipeline in later labs. Imputing before the split would allow test-set statistics to influence training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-009",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_features = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
    "infinite_count = int(np.isinf(numeric_features.to_numpy(dtype=float)).sum())\n",
    "missing_summary = numeric_features.isna().mean().sort_values(ascending=False).rename(\"missing_rate\")\n",
    "duplicate_count = int(df.duplicated().sum())\n",
    "constant_features = numeric_features.nunique(dropna=True).loc[lambda values: values <= 1].index.tolist()\n",
    "\n",
    "suspicious_tokens = (\"label\", \"class\", \"target\", \"source\", \"filename\", \"hash\", \"md5\", \"sha\")\n",
    "suspicious_columns = [\n",
    "    column for column in feature_cols\n",
    "    if any(token in column.lower() for token in suspicious_tokens)\n",
    "]\n",
    "\n",
    "print(\"Exact duplicate rows:\", duplicate_count)\n",
    "print(\"Infinite numeric values:\", infinite_count)\n",
    "print(\"Constant candidate features:\", len(constant_features))\n",
    "print(\"Suspicious feature names:\", suspicious_columns[:20])\n",
    "display(missing_summary.head(10).to_frame())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-010",
   "metadata": {},
   "source": [
    "## 4. Write the dataset risk statement\n",
    "\n",
    "Record three facts before modeling:\n",
    "\n",
    "1. The positive class is the majority class.\n",
    "2. Benign and malware rows come from different source collections.\n",
    "3. A random split can measure interpolation within this historical table, but not performance on future malware campaigns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-011",
   "metadata": {},
   "source": [
    "## Self-check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-012",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert {\"source\", \"label\"}.issubset(df.columns)\n",
    "assert len(feature_cols) > 0\n",
    "assert set(df[\"label\"].dropna().unique()) == {0, 1}\n",
    "assert set(feature_cols).isdisjoint({\"source\", \"label\"})\n",
    "assert class_counts.loc[1] > class_counts.loc[0]\n",
    "assert len(missing_summary) == len(feature_cols)\n",
    "assert infinite_count == 0\n",
    "print(\"✓ Lab 1 completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-013",
   "metadata": {},
   "source": [
    "## Suggested answers\n",
    "\n",
    "1. **Why exclude `source`?** The source is strongly associated with the label. A model could learn the collection process instead of malware behavior.\n",
    "2. **Why delay imputation?** The imputer must learn medians from training data only; otherwise test information leaks into training.\n",
    "3. **Why is a random split limited?** Samples from the same historical collections appear on both sides of the split, so the test set does not represent future drift or a new collection process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c3b5e17-8d80-4c4b-a68f-7d6e085b04f0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a20ec3d1-331a-46ab-8e18-f3cd73ff9fc3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}