CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,271 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-001",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 5 — Lab 4: Security Metrics and Error Costs\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** false negatives, class weights, validation threshold, test discipline\n",
|
||||
"\n",
|
||||
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-002",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- quantify false-negative and false-positive costs;\n",
|
||||
"- compare class-weight policies on validation data;\n",
|
||||
"- select a probability threshold without inspecting the test set;\n",
|
||||
"- evaluate the chosen policy once on held-out test data;\n",
|
||||
"- describe the operational trade-off between missed malware and analyst workload."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-003",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
|
||||
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"from sklearn.dummy import DummyClassifier\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.tree import DecisionTreeClassifier\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def security_metrics(y_true, y_pred):\n",
|
||||
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
|
||||
" return {\n",
|
||||
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
|
||||
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"false_positives\": int(fp),\n",
|
||||
" \"false_negatives\": int(fn),\n",
|
||||
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
|
||||
" }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-004",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
|
||||
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
|
||||
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
|
||||
"y = df[\"label\"].astype(int)\n",
|
||||
"\n",
|
||||
"X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
|
||||
" X, y, test_size=0.20, stratify=y, random_state=42\n",
|
||||
")\n",
|
||||
"X_train, X_valid, y_train, y_valid = train_test_split(\n",
|
||||
" X_train_full, y_train_full, test_size=0.25, stratify=y_train_full, random_state=42\n",
|
||||
")\n",
|
||||
"print(\"Train/validation/test:\", len(X_train), len(X_valid), len(X_test))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-005",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Compare class-weight policies on validation data\n",
|
||||
"\n",
|
||||
"Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-006",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"weight_policies = {\n",
|
||||
" \"unweighted\": None,\n",
|
||||
" \"malware_x2\": {0: 1, 1: 2},\n",
|
||||
" \"malware_x4\": {0: 1, 1: 4},\n",
|
||||
"}\n",
|
||||
"validation_rows = []\n",
|
||||
"fitted_candidates = {}\n",
|
||||
"\n",
|
||||
"for name, weights in weight_policies.items():\n",
|
||||
" model = Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
|
||||
" (\"tree\", DecisionTreeClassifier(\n",
|
||||
" max_depth=4, min_samples_leaf=10, class_weight=weights, random_state=42\n",
|
||||
" )),\n",
|
||||
" ])\n",
|
||||
" model.fit(X_train, y_train)\n",
|
||||
" predictions = model.predict(X_valid)\n",
|
||||
" validation_rows.append({\"policy\": name, **security_metrics(y_valid, predictions)})\n",
|
||||
" fitted_candidates[name] = model\n",
|
||||
"\n",
|
||||
"validation_results = pd.DataFrame(validation_rows).set_index(\"policy\")\n",
|
||||
"display(validation_results.round(4))\n",
|
||||
"selected_policy = validation_results.sort_values(\n",
|
||||
" [\"malware_recall\", \"false_positive_rate\", \"malware_f1\"],\n",
|
||||
" ascending=[False, True, False],\n",
|
||||
").index[0]\n",
|
||||
"selected_model = fitted_candidates[selected_policy]\n",
|
||||
"print(\"Selected class-weight policy:\", selected_policy)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-007",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Select a validation threshold\n",
|
||||
"\n",
|
||||
"Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-008",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"validation_probabilities = selected_model.predict_proba(X_valid)[:, 1]\n",
|
||||
"threshold_rows = []\n",
|
||||
"for threshold in np.arange(0.20, 0.81, 0.05):\n",
|
||||
" predictions = (validation_probabilities >= threshold).astype(int)\n",
|
||||
" threshold_rows.append({\"threshold\": round(float(threshold), 2), **security_metrics(y_valid, predictions)})\n",
|
||||
"\n",
|
||||
"threshold_results = pd.DataFrame(threshold_rows).set_index(\"threshold\")\n",
|
||||
"eligible = threshold_results[threshold_results[\"malware_recall\"] >= 0.98]\n",
|
||||
"pool = eligible if not eligible.empty else threshold_results.loc[[threshold_results[\"malware_recall\"].idxmax()]]\n",
|
||||
"selected_threshold = float(pool.sort_values(\n",
|
||||
" [\"false_positive_rate\", \"malware_f1\"], ascending=[True, False]\n",
|
||||
").index[0])\n",
|
||||
"display(threshold_results.round(4))\n",
|
||||
"print(\"Selected threshold:\", selected_threshold)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-009",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Refit on all training data and evaluate once on the test set\n",
|
||||
"\n",
|
||||
"The test set is used only after all policy decisions are fixed."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-010",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"final_model = Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
|
||||
" (\"tree\", DecisionTreeClassifier(\n",
|
||||
" max_depth=4,\n",
|
||||
" min_samples_leaf=10,\n",
|
||||
" class_weight=weight_policies[selected_policy],\n",
|
||||
" random_state=42,\n",
|
||||
" )),\n",
|
||||
"])\n",
|
||||
"final_model.fit(X_train_full, y_train_full)\n",
|
||||
"test_probabilities = final_model.predict_proba(X_test)[:, 1]\n",
|
||||
"default_predictions = (test_probabilities >= 0.50).astype(int)\n",
|
||||
"tuned_predictions = (test_probabilities >= selected_threshold).astype(int)\n",
|
||||
"\n",
|
||||
"test_results = pd.DataFrame([\n",
|
||||
" security_metrics(y_test, default_predictions),\n",
|
||||
" security_metrics(y_test, tuned_predictions),\n",
|
||||
"], index=[\"threshold_0.50\", f\"threshold_{selected_threshold:.2f}\"])\n",
|
||||
"display(test_results.round(4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-011",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-012",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert set(validation_results.index) == set(weight_policies)\n",
|
||||
"assert 0.20 <= selected_threshold <= 0.80\n",
|
||||
"assert len(test_probabilities) == len(y_test)\n",
|
||||
"assert set(tuned_predictions).issubset({0, 1})\n",
|
||||
"assert test_results.shape[0] == 2\n",
|
||||
"assert final_model.named_steps[\"tree\"].get_depth() <= 4\n",
|
||||
"print(\"✓ Lab 4 completed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-013",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Suggested answers\n",
|
||||
"\n",
|
||||
"1. **Why tune on validation data?** Repeatedly choosing settings from test results turns the test set into training information.\n",
|
||||
"2. **Why prioritize malware recall?** A false negative allows a malicious file to pass undetected. However, maximizing recall without an FPR constraint can overwhelm analysts.\n",
|
||||
"3. **Is the chosen threshold universal?** No. It depends on prevalence, feature drift, collection methods, and operational costs."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user