CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,218 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-001",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"**Topics:** feature families, distributions, effect size, responsible EDA\n",
|
||||
"\n",
|
||||
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-002",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- summarize numeric malware features by class;\n",
|
||||
"- identify interpretable feature-name families;\n",
|
||||
"- compare benign and malware distributions with robust statistics;\n",
|
||||
"- distinguish exploratory association from causal evidence;\n",
|
||||
"- avoid using the held-out test set for model-driven feature selection."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-003",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
|
||||
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-004",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 1: load the prepared dataset and define X, y, and feature_cols.\n",
|
||||
"df = None\n",
|
||||
"feature_cols = None\n",
|
||||
"X = None\n",
|
||||
"y = None\n",
|
||||
"print(\"Rows:\", None, \"Features:\", None)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-005",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Inspect feature-name families\n",
|
||||
"\n",
|
||||
"Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-006",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"family_tokens = {\n",
|
||||
" \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
|
||||
" \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
|
||||
" \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
|
||||
" \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# TODO 2: collect up to ten feature-name examples for each family.\n",
|
||||
"family_examples = None\n",
|
||||
"family_examples"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-007",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Build a robust class-comparison profile\n",
|
||||
"\n",
|
||||
"Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-008",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"benign = X.loc[y == 0]\n",
|
||||
"malware = X.loc[y == 1]\n",
|
||||
"\n",
|
||||
"# TODO 3: calculate the IQR for every feature and replace zero IQR values with NaN.\n",
|
||||
"iqr = None\n",
|
||||
"\n",
|
||||
"# TODO 4: build feature_profile with missing rate, unique count, and class medians.\n",
|
||||
"feature_profile = None\n",
|
||||
"\n",
|
||||
"# TODO 5: calculate the absolute median gap divided by IQR, then select the top eight names.\n",
|
||||
"top_features = None\n",
|
||||
"display(feature_profile.loc[top_features])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-009",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Visualize class medians for the strongest exploratory differences"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-010",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO 6: create a horizontal grouped bar chart of benign and malware medians.\n",
|
||||
"median_plot = None\n",
|
||||
"ax = None\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-011",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Interpretation warning\n",
|
||||
"\n",
|
||||
"A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-012",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-013",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert set(X.columns) == set(feature_cols)\n",
|
||||
"assert set(y.unique()) == {0, 1}\n",
|
||||
"assert len(feature_profile) == len(feature_cols)\n",
|
||||
"assert len(top_features) == 8\n",
|
||||
"assert set(top_features).issubset(feature_cols)\n",
|
||||
"assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
|
||||
"print(\"✓ Lab 2 completed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-014",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reflection questions\n",
|
||||
"\n",
|
||||
"1. Why are medians and IQR useful for security features?\n",
|
||||
"2. Can a large standardized median gap prove that a feature causes malware behavior?\n",
|
||||
"3. Why should this full-dataset ranking not be used directly to optimize the final model?"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user