CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,235 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-001",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** tree rules, complexity, feature importance, train-test gap\n",
|
||||
"\n",
|
||||
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-002",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- compare bounded and unrestricted trees;\n",
|
||||
"- quantify depth, number of leaves, and train-test performance gaps;\n",
|
||||
"- visualize an explainable tree;\n",
|
||||
"- export root-to-leaf logic as text rules;\n",
|
||||
"- interpret feature importance cautiously."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-003",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
|
||||
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"from sklearn.dummy import DummyClassifier\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.tree import DecisionTreeClassifier\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"from sklearn.tree import export_text, plot_tree\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def security_metrics(y_true, y_pred):\n",
|
||||
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
|
||||
" return {\n",
|
||||
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
|
||||
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
|
||||
" \"false_positives\": int(fp),\n",
|
||||
" \"false_negatives\": int(fn),\n",
|
||||
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
|
||||
" }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-004",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 1: load data, define X/y, and make an 80/20 stratified split.\n",
|
||||
"df = None\n",
|
||||
"feature_cols = None\n",
|
||||
"X = None\n",
|
||||
"y = None\n",
|
||||
"X_train, X_test, y_train, y_test = None, None, None, None\n",
|
||||
"\n",
|
||||
"# TODO 2: implement make_tree and fit a bounded tree (depth 4, leaf 10)\n",
|
||||
"# and an unrestricted tree (no maximum depth, leaf 1).\n",
|
||||
"def make_tree(max_depth, min_samples_leaf):\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"bounded_tree = None\n",
|
||||
"unrestricted_tree = None"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-005",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Compare model complexity and generalization"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-006",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 3: create complexity_comparison with depth, leaves, train/test F1,\n",
|
||||
"# F1 gap, test recall, and test false-positive rate for both trees.\n",
|
||||
"complexity_comparison = None\n",
|
||||
"display(complexity_comparison)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-007",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Visualize and export the bounded tree"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-008",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 4: obtain transformed feature names, plot the bounded tree, and\n",
|
||||
"# export its rules as text with max_depth=4.\n",
|
||||
"imputer = None\n",
|
||||
"estimator = None\n",
|
||||
"transformed_names = None\n",
|
||||
"rule_text = None\n",
|
||||
"print(rule_text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-009",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Inspect impurity-based feature importance"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-010",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 5: create a descending importance Series using transformed names,\n",
|
||||
"# display the top 12, and plot a horizontal bar chart.\n",
|
||||
"importance = None\n",
|
||||
"top_importance = None\n",
|
||||
"display(top_importance)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-011",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Interpretation warning\n",
|
||||
"\n",
|
||||
"A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-012",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-013",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert bounded_tree.named_steps[\"tree\"].get_depth() <= 4\n",
|
||||
"assert unrestricted_tree.named_steps[\"tree\"].get_depth() >= bounded_tree.named_steps[\"tree\"].get_depth()\n",
|
||||
"assert complexity_comparison.loc[\"unrestricted\", \"leaves\"] >= complexity_comparison.loc[\"bounded\", \"leaves\"]\n",
|
||||
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
|
||||
"assert np.isclose(importance.sum(), 1.0)\n",
|
||||
"assert len(top_importance) <= 12\n",
|
||||
"print(\"✓ Lab 5 completed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-014",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reflection questions\n",
|
||||
"\n",
|
||||
"1. Which values in the comparison table indicate overfitting?\n",
|
||||
"2. Translate two root-to-malware paths into `IF ... AND ... THEN malware` rules.\n",
|
||||
"3. Why must feature importance not be interpreted as causal evidence?"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user