CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,227 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-001",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** feature families, distributions, effect size, responsible EDA\n",
|
||||
"\n",
|
||||
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-002",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- summarize numeric malware features by class;\n",
|
||||
"- identify interpretable feature-name families;\n",
|
||||
"- compare benign and malware distributions with robust statistics;\n",
|
||||
"- distinguish exploratory association from causal evidence;\n",
|
||||
"- avoid using the held-out test set for model-driven feature selection."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-003",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
|
||||
"\n",
|
||||
"if not DATA_PATH.exists():\n",
|
||||
" raise FileNotFoundError(\n",
|
||||
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
|
||||
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-004",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
|
||||
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
|
||||
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
|
||||
"y = df[\"label\"].astype(int)\n",
|
||||
"print(\"Rows:\", len(df), \"Features:\", len(feature_cols))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-005",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Inspect feature-name families\n",
|
||||
"\n",
|
||||
"Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-006",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"family_tokens = {\n",
|
||||
" \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
|
||||
" \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
|
||||
" \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
|
||||
" \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"family_examples = {}\n",
|
||||
"for family, tokens in family_tokens.items():\n",
|
||||
" matches = [name for name in feature_cols if any(token in name.lower() for token in tokens)]\n",
|
||||
" family_examples[family] = matches[:10]\n",
|
||||
"family_examples"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-007",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Build a robust class-comparison profile\n",
|
||||
"\n",
|
||||
"Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-008",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"benign = X.loc[y == 0]\n",
|
||||
"malware = X.loc[y == 1]\n",
|
||||
"q75 = X.quantile(0.75)\n",
|
||||
"q25 = X.quantile(0.25)\n",
|
||||
"iqr = (q75 - q25).replace(0, np.nan)\n",
|
||||
"\n",
|
||||
"feature_profile = pd.DataFrame({\n",
|
||||
" \"missing_rate\": X.isna().mean(),\n",
|
||||
" \"unique_values\": X.nunique(dropna=True),\n",
|
||||
" \"benign_median\": benign.median(),\n",
|
||||
" \"malware_median\": malware.median(),\n",
|
||||
"})\n",
|
||||
"feature_profile[\"standardized_median_gap\"] = (\n",
|
||||
" (feature_profile[\"malware_median\"] - feature_profile[\"benign_median\"]).abs() / iqr\n",
|
||||
")\n",
|
||||
"feature_profile = feature_profile.replace([np.inf, -np.inf], np.nan)\n",
|
||||
"top_features = feature_profile[\"standardized_median_gap\"].nlargest(8).index.tolist()\n",
|
||||
"display(feature_profile.loc[top_features])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-009",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Visualize class medians for the strongest exploratory differences"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-010",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"median_plot = feature_profile.loc[top_features, [\"benign_median\", \"malware_median\"]]\n",
|
||||
"ax = median_plot.plot(kind=\"barh\", figsize=(10, 6))\n",
|
||||
"ax.set_title(\"Class medians for exploratory high-gap features\")\n",
|
||||
"ax.set_xlabel(\"Median feature value\")\n",
|
||||
"ax.set_ylabel(\"Feature\")\n",
|
||||
"plt.tight_layout()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-011",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Interpretation warning\n",
|
||||
"\n",
|
||||
"A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-012",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "cell-013",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert set(X.columns) == set(feature_cols)\n",
|
||||
"assert set(y.unique()) == {0, 1}\n",
|
||||
"assert len(feature_profile) == len(feature_cols)\n",
|
||||
"assert len(top_features) == 8\n",
|
||||
"assert set(top_features).issubset(feature_cols)\n",
|
||||
"assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
|
||||
"print(\"✓ Lab 2 completed\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cell-014",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Suggested answers\n",
|
||||
"\n",
|
||||
"1. **Why use medians and IQR?** They are less sensitive to extreme values and skew than means and standard deviations.\n",
|
||||
"2. **Can the ranking prove causation?** No. It identifies associations in this particular collection.\n",
|
||||
"3. **Why not choose final features from the full table?** Doing so lets test-set patterns influence model design and makes the final score optimistic."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user