labs/Lab-3/Lab-3-Practical/05_Class_Imbalance_Experiment_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "24b7696f",
   "metadata": {},
   "source": [
    "# Module 3 — Lab 5: A Controlled Class-Imbalance Experiment\n",
    "\n",
    "**Topics:** class imbalance, class weights, controlled comparison, metric trade-offs\n",
    "\n",
    "> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ab7f5aa",
   "metadata": {},
   "source": [
    "## Learning objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- measure the class imbalance in a security dataset;\n",
    "- compare two runs of the same model on the same split;\n",
    "- use `class_weight=\"balanced\"` without changing the rest of the workflow;\n",
    "- select a model configuration using attack precision, recall, and F1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "776b4b02",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
    "RANDOM_STATE = 42\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7386484c",
   "metadata": {},
   "source": [
    "## 1. Prepare one fixed experiment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61510ed9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
    "df.columns = df.columns.str.strip()\n",
    "df = df.replace([np.inf, -np.inf], np.nan)\n",
    "\n",
    "X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
    "y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
    ")\n",
    "\n",
    "# TODO 1: calculate benign_count, attack_count, and imbalance_ratio\n",
    "benign_count = None\n",
    "attack_count = None\n",
    "imbalance_ratio = None\n",
    "\n",
    "print(f\"BENIGN={benign_count}, ATTACK={attack_count}\")\n",
    "print(f\"BENIGN-to-ATTACK ratio: {imbalance_ratio:.2f}:1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcec4c13",
   "metadata": {},
   "source": [
    "## 2. Define a reusable training function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b143e2da",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_and_measure(class_weight):\n",
    "    # TODO 2: create the imputer + Random Forest pipeline\n",
    "    model = None\n",
    "\n",
    "    # TODO 3: fit, predict, and return a metrics dictionary\n",
    "    return None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e81c919c",
   "metadata": {},
   "source": [
    "## 3. Run the controlled comparison\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5aa0bbb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the data split and all hyperparameters fixed.\n",
    "# TODO 4: run the baseline with class_weight=None\n",
    "baseline_metrics = None\n",
    "\n",
    "# TODO 5: run the weighted model with class_weight=\"balanced\"\n",
    "balanced_metrics = None\n",
    "\n",
    "# TODO 6: create comparison_df with Baseline and Balanced rows\n",
    "comparison_df = None\n",
    "\n",
    "comparison_df.round(3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d9d295d",
   "metadata": {},
   "source": [
    "## 4. Make an evidence-based recommendation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e08131ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO 7: select the configuration with the higher attack F1 score\n",
    "recommended_configuration = None\n",
    "\n",
    "print(\"Recommended configuration:\", recommended_configuration)\n",
    "print(comparison_df.loc[recommended_configuration].round(3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6831bc3",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00a29862",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert benign_count > attack_count\n",
    "assert imbalance_ratio > 1\n",
    "assert list(comparison_df.index) == [\"Baseline\", \"Balanced\"]\n",
    "assert {\"precision\", \"recall\", \"f1\", \"false_positives\", \"false_negatives\"}.issubset(comparison_df.columns)\n",
    "assert comparison_df[[\"precision\", \"recall\", \"f1\"]].apply(lambda col: col.between(0, 1).all()).all()\n",
    "assert recommended_configuration in {\"Baseline\", \"Balanced\"}\n",
    "print(\"✓ Lab 5 completed\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79d33904",
   "metadata": {},
   "source": [
    "## Analyst questions\n",
    "\n",
    "1. Did class weighting improve attack recall in this run?\n",
    "2. What happened to false positives and false negatives?\n",
    "3. Would you choose the same configuration if analyst time were extremely limited?\n",
    "4. Why is changing only one variable important in a controlled experiment?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}