mikrotik/labs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Browse Source
This commit is contained in:
mikrotik
2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Lab-5/Lab-5-Practical/01_Malware_Dataset_Audit_Lab.ipynb
+222
View File
@@ -0,0 +1,222 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 1: Auditing a Malware Feature Dataset\n",
"\n",
"**Topics:** provenance, schema, imbalance, missing values, leakage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load a prepared table of static and dynamic malware features safely;\n",
"- distinguish identifiers, provenance fields, features, and the binary label;\n",
"- audit class imbalance, source composition, missing values, infinite values, and duplicates;\n",
"- identify columns that must not enter a predictive model;\n",
"- document dataset limitations before modeling.\n",
"\n",
"> **Safety boundary:** This practical uses numeric feature tables only. Do not download, open, or execute malware samples."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load the prepared table\n",
"\n",
"The preparation utility adds two non-feature columns: `source` records provenance and `label` uses `0 = benign`, `1 = malware`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data/malware_features.csv into df.\n",
"df = None\n",
"\n",
"# TODO 2: verify that source and label exist, then create feature_cols.\n",
"required_columns = {\"source\", \"label\"}\n",
"feature_cols = None\n",
"\n",
"print(\"Dataset shape:\", None)\n",
"print(\"Number of candidate features:\", None)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit labels and provenance\n",
"\n",
"`source` is useful for detecting collection bias, but it must never be used as a predictor. In this dataset, collection source and class are closely related."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-007",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: calculate class counts and class proportions.\n",
"class_counts = None\n",
"class_proportions = None\n",
"\n",
"# TODO 4: create a source-by-label cross-tabulation with margins.\n",
"source_by_label = None\n",
"\n",
"display(pd.DataFrame({\"count\": class_counts, \"proportion\": class_proportions}))\n",
"display(source_by_label)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit data quality and leakage risks\n",
"\n",
"Missing values will be imputed inside a training pipeline in later labs. Imputing before the split would allow test-set statistics to influence training."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-009",
"metadata": {},
"outputs": [],
"source": [
"numeric_features = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"\n",
"# TODO 5: count infinite numeric values and exact duplicate rows.\n",
"infinite_count = None\n",
"duplicate_count = None\n",
"\n",
"# TODO 6: calculate per-feature missing rates and find constant features.\n",
"missing_summary = None\n",
"constant_features = None\n",
"\n",
"# TODO 7: find feature names containing possible label, source, filename, or hash tokens.\n",
"suspicious_tokens = (\"label\", \"class\", \"target\", \"source\", \"filename\", \"hash\", \"md5\", \"sha\")\n",
"suspicious_columns = None\n",
"\n",
"print(\"Exact duplicate rows:\", duplicate_count)\n",
"print(\"Infinite numeric values:\", infinite_count)\n",
"print(\"Constant candidate features:\", len(constant_features))\n",
"print(\"Suspicious feature names:\", suspicious_columns[:20])\n",
"display(missing_summary.head(10).to_frame())"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## 4. Write the dataset risk statement\n",
"\n",
"Record three facts before modeling:\n",
"\n",
"1. The positive class is the majority class.\n",
"2. Benign and malware rows come from different source collections.\n",
"3. A random split can measure interpolation within this historical table, but not performance on future malware campaigns."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert {\"source\", \"label\"}.issubset(df.columns)\n",
"assert len(feature_cols) > 0\n",
"assert set(df[\"label\"].dropna().unique()) == {0, 1}\n",
"assert set(feature_cols).isdisjoint({\"source\", \"label\"})\n",
"assert class_counts.loc[1] > class_counts.loc[0]\n",
"assert len(missing_summary) == len(feature_cols)\n",
"assert infinite_count == 0\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must `source` be excluded from the model?\n",
"2. Why should missing-value imputation happen after the train-test split?\n",
"3. Why is a random split not sufficient evidence for production deployment?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/02_Malware_Feature_Exploration_Lab.ipynb
+218
View File
@@ -0,0 +1,218 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
"\n",
" \n",
"**Topics:** feature families, distributions, effect size, responsible EDA\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- summarize numeric malware features by class;\n",
"- identify interpretable feature-name families;\n",
"- compare benign and malware distributions with robust statistics;\n",
"- distinguish exploratory association from causal evidence;\n",
"- avoid using the held-out test set for model-driven feature selection."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the prepared dataset and define X, y, and feature_cols.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"print(\"Rows:\", None, \"Features:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect feature-name families\n",
"\n",
"Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"family_tokens = {\n",
" \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
" \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
" \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
" \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
"}\n",
"\n",
"# TODO 2: collect up to ten feature-name examples for each family.\n",
"family_examples = None\n",
"family_examples"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build a robust class-comparison profile\n",
"\n",
"Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"benign = X.loc[y == 0]\n",
"malware = X.loc[y == 1]\n",
"\n",
"# TODO 3: calculate the IQR for every feature and replace zero IQR values with NaN.\n",
"iqr = None\n",
"\n",
"# TODO 4: build feature_profile with missing rate, unique count, and class medians.\n",
"feature_profile = None\n",
"\n",
"# TODO 5: calculate the absolute median gap divided by IQR, then select the top eight names.\n",
"top_features = None\n",
"display(feature_profile.loc[top_features])"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Visualize class medians for the strongest exploratory differences"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: create a horizontal grouped bar chart of benign and malware medians.\n",
"median_plot = None\n",
"ax = None\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert set(X.columns) == set(feature_cols)\n",
"assert set(y.unique()) == {0, 1}\n",
"assert len(feature_profile) == len(feature_cols)\n",
"assert len(top_features) == 8\n",
"assert set(top_features).issubset(feature_cols)\n",
"assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why are medians and IQR useful for security features?\n",
"2. Can a large standardized median gap prove that a feature causes malware behavior?\n",
"3. Why should this full-dataset ranking not be used directly to optimize the final model?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/03_Decision_Tree_Baseline_Lab.ipynb
+203
View File
@@ -0,0 +1,203 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 3: Building a Decision Tree Baseline\n",
"\n",
"\n",
"**Topics:** stratified split, dummy baseline, bounded tree, leakage-safe pipeline\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- create a reproducible stratified train-test split;\n",
"- fit missing-value imputation on training data only;\n",
"- compare a bounded Decision Tree with a dummy baseline;\n",
"- evaluate malware precision, recall, F1, false positives, and false negatives;\n",
"- explain why trees do not require feature scaling."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the data, exclude source and label from X, and create integer y.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"\n",
"# TODO 2: make an 80/20 stratified split with random_state=42.\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"print(\"Train shape:\", None, \"Test shape:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Establish a non-learning baseline\n",
"\n",
"Because malware is the majority class, a most-frequent classifier can achieve high accuracy while providing little operational value."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: fit a most-frequent DummyClassifier and calculate security metrics.\n",
"dummy = None\n",
"dummy_predictions = None\n",
"dummy_metrics = None\n",
"pd.Series(dummy_metrics, name=\"dummy\")"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Train a bounded Decision Tree\n",
"\n",
"The imputer is inside the pipeline, so its medians are learned from training rows only. A shallow tree is intentionally chosen for readability and to reduce overfitting."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: create a pipeline with median imputation and a Decision Tree.\n",
"# Use max_depth=4, min_samples_leaf=10, and random_state=42.\n",
"tree_pipeline = None\n",
"\n",
"# TODO 5: fit the pipeline, predict the test set, and calculate security metrics.\n",
"tree_predictions = None\n",
"tree_metrics = None\n",
"\n",
"comparison = None\n",
"display(comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"assert X_train.shape[0] + X_test.shape[0] == len(df)\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.01\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 4\n",
"assert set(comparison.index) == {\"Dummy\", \"Decision Tree\"}\n",
"assert comparison.loc[\"Decision Tree\", \"malware_f1\"] >= comparison.loc[\"Dummy\", \"malware_f1\"]\n",
"print(\"✓ Lab 3 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why can the dummy model have deceptively high accuracy?\n",
"2. Why does a Decision Tree usually not require feature scaling?\n",
"3. What is the security and interpretability benefit of limiting tree depth?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/04_Security_Metrics_Error_Costs_Lab.ipynb
+242
View File
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 4: Security Metrics and Error Costs\n",
"\n",
" \n",
"**Topics:** false negatives, class weights, validation threshold, test discipline\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- quantify false-negative and false-positive costs;\n",
"- compare class-weight policies on validation data;\n",
"- select a probability threshold without inspecting the test set;\n",
"- evaluate the chosen policy once on held-out test data;\n",
"- describe the operational trade-off between missed malware and analyst workload."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load X and y, create a 20% stratified test set, then split the\n",
"# remaining data into 75% train and 25% validation using random_state=42.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train_full, X_test, y_train_full, y_test = None, None, None, None\n",
"X_train, X_valid, y_train, y_valid = None, None, None, None\n",
"print(\"Train/validation/test:\", None, None, None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare class-weight policies on validation data\n",
"\n",
"Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"weight_policies = {\n",
" \"unweighted\": None,\n",
" \"malware_x2\": {0: 1, 1: 2},\n",
" \"malware_x4\": {0: 1, 1: 4},\n",
"}\n",
"\n",
"# TODO 2: fit one depth-4 tree pipeline per weight policy and collect\n",
"# validation security_metrics in validation_results.\n",
"validation_results = None\n",
"fitted_candidates = None\n",
"display(validation_results)\n",
"\n",
"# TODO 3: select highest recall, breaking ties by lower FPR then higher F1.\n",
"selected_policy = None\n",
"selected_model = None\n",
"print(\"Selected class-weight policy:\", selected_policy)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a validation threshold\n",
"\n",
"Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: obtain validation malware probabilities and evaluate thresholds\n",
"# from 0.20 through 0.80 in steps of 0.05.\n",
"validation_probabilities = None\n",
"threshold_results = None\n",
"\n",
"# TODO 5: among thresholds with recall >= 0.98, select the lowest FPR.\n",
"# If none qualify, use a threshold with maximum recall.\n",
"selected_threshold = None\n",
"display(threshold_results)\n",
"print(\"Selected threshold:\", selected_threshold)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Refit on all training data and evaluate once on the test set\n",
"\n",
"The test set is used only after all policy decisions are fixed."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: refit the selected policy on X_train_full/y_train_full.\n",
"final_model = None\n",
"\n",
"# TODO 7: compare threshold 0.50 and selected_threshold on the test set.\n",
"test_probabilities = None\n",
"default_predictions = None\n",
"tuned_predictions = None\n",
"test_results = None\n",
"display(test_results)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(validation_results.index) == set(weight_policies)\n",
"assert 0.20 <= selected_threshold <= 0.80\n",
"assert len(test_probabilities) == len(y_test)\n",
"assert set(tuned_predictions).issubset({0, 1})\n",
"assert test_results.shape[0] == 2\n",
"assert final_model.named_steps[\"tree\"].get_depth() <= 4\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must model and threshold choices use validation data rather than test data?\n",
"2. What operational harm can false negatives and false positives cause?\n",
"3. Why should the selected threshold be revalidated after deployment conditions change?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/05_Tree_Interpretation_Overfitting_Lab.ipynb
+235
View File
@@ -0,0 +1,235 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting\n",
"\n",
"\n",
"**Topics:** tree rules, complexity, feature importance, train-test gap\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare bounded and unrestricted trees;\n",
"- quantify depth, number of leaves, and train-test performance gaps;\n",
"- visualize an explainable tree;\n",
"- export root-to-leaf logic as text rules;\n",
"- interpret feature importance cautiously."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.tree import export_text, plot_tree\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, define X/y, and make an 80/20 stratified split.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"# TODO 2: implement make_tree and fit a bounded tree (depth 4, leaf 10)\n",
"# and an unrestricted tree (no maximum depth, leaf 1).\n",
"def make_tree(max_depth, min_samples_leaf):\n",
" pass\n",
"\n",
"bounded_tree = None\n",
"unrestricted_tree = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare model complexity and generalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create complexity_comparison with depth, leaves, train/test F1,\n",
"# F1 gap, test recall, and test false-positive rate for both trees.\n",
"complexity_comparison = None\n",
"display(complexity_comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Visualize and export the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: obtain transformed feature names, plot the bounded tree, and\n",
"# export its rules as text with max_depth=4.\n",
"imputer = None\n",
"estimator = None\n",
"transformed_names = None\n",
"rule_text = None\n",
"print(rule_text)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Inspect impurity-based feature importance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: create a descending importance Series using transformed names,\n",
"# display the top 12, and plot a horizontal bar chart.\n",
"importance = None\n",
"top_importance = None\n",
"display(top_importance)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert bounded_tree.named_steps[\"tree\"].get_depth() <= 4\n",
"assert unrestricted_tree.named_steps[\"tree\"].get_depth() >= bounded_tree.named_steps[\"tree\"].get_depth()\n",
"assert complexity_comparison.loc[\"unrestricted\", \"leaves\"] >= complexity_comparison.loc[\"bounded\", \"leaves\"]\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Which values in the comparison table indicate overfitting?\n",
"2. Translate two root-to-malware paths into `IF ... AND ... THEN malware` rules.\n",
"3. Why must feature importance not be interpreted as causal evidence?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/06_Capstone_Malware_Detection_Lab.ipynb
+275
View File
@@ -0,0 +1,275 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow\n",
"\n",
"\n",
"**Topics:** end-to-end workflow, model comparison, error evidence, analyst recommendation\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/capstone_metrics.csv`\n",
"- `output/capstone_errors.csv`\n",
"- `output/capstone_tree_rules.txt`\n",
"- `output/capstone_confusion_matrix.png`\n",
"- `output/capstone_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.tree import export_text\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the dataset, exclude source/label from X, and create an\n",
"# 80/20 stratified split that also retains source_test for error auditing.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train, X_test, y_train, y_test, source_train, source_test = (None,) * 6\n",
"print(\"Train/test rows:\", None, None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Build and compare candidate models"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: define Dummy, ExplainableTree, DeepTree, and RandomForest.\n",
"# Keep ExplainableTree at max_depth=4 and min_samples_leaf=10.\n",
"models = None\n",
"\n",
"# TODO 3: fit every model, save predictions/probabilities, and build a\n",
"# metrics table with security_metrics.\n",
"metric_rows = None\n",
"predictions = None\n",
"probabilities = None\n",
"metrics_table = None\n",
"display(metrics_table)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a model with an explicit decision rule\n",
"\n",
"Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: implement the documented selection rule.\n",
"eligible_metrics = None\n",
"best_recall = None\n",
"best_fpr = None\n",
"explainable_ok = None\n",
"selected_name = None\n",
"selected_model = None\n",
"selected_predictions = None\n",
"print(\"Selected model:\", selected_name)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Create error evidence and a confusion matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: create error_table with source, true label, prediction,\n",
"# malware probability, and error_type. Keep mistakes in errors_only.\n",
"error_table = None\n",
"errors_only = None\n",
"display(errors_only[\"error_type\"].value_counts())\n",
"\n",
"# TODO 6: plot and save the selected model's confusion matrix."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export rules and write the analyst recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: export the ExplainableTree rules even if another model was selected.\n",
"tree_rules = None\n",
"\n",
"# TODO 8: write an English analyst summary covering the recommendation,\n",
"# metrics, FP/FN costs, data limitations, and deployment safeguards.\n",
"analyst_summary = None\n",
"\n",
"# TODO 9: save the metrics, errors, rules, and analyst summary.\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_errors.csv\",\n",
" \"capstone_tree_rules.txt\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_analyst_summary.md\",\n",
"}\n",
"assert set(models) == {\"Dummy\", \"ExplainableTree\", \"DeepTree\", \"RandomForest\"}\n",
"assert selected_name in set(models) - {\"Dummy\"}\n",
"assert len(error_table) == len(y_test)\n",
"assert set(errors_only[\"error_type\"].unique()).issubset({\"false_negative\", \"false_positive\"})\n",
"assert isinstance(tree_rules, str) and \"class:\" in tree_rules\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom.\n",
"2. Confirm that all five output files exist.\n",
"3. Verify that the recommendation is supported by the metrics table.\n",
"4. Explain at least one false negative and one false positive operationally.\n",
"5. State why this random-split result is not a production guarantee."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/data/README.md
+79
View File
@@ -0,0 +1,79 @@
# Dataset setup — UCI Malware Static and Dynamic Features
All six notebooks use pre-extracted tabular features from the official UCI dataset **Malware Static and Dynamic Features VxHeaven and Virus Total**.
## Download
1. Open the [UCI dataset page](https://archive.ics.uci.edu/dataset/541/malware%2Bstatic%2Band%2Bdynamic%2Bfeatures%2Bvxheaven%2Band%2Bvirus%2Btotal).
2. Select **Download** and extract the archive.
3. Create `data/raw/` if it does not already exist.
4. Copy these three files into `data/raw/`:
```text
staDynBenignLab.csv
staDynVxHeaven2698Lab.csv
staDynVt2955Lab.csv
```
The expected layout is:
```text
Practical-5/
└── data/
├── prepare_dataset.py
└── raw/
├── staDynBenignLab.csv
├── staDynVxHeaven2698Lab.csv
└── staDynVt2955Lab.csv
```
## Prepare the classroom table
From the `Practical-5` directory, run:
```bash
python data/prepare_dataset.py
```
The utility:
- finds the three required files under `data/raw/`;
- normalizes column names and adds deterministic occurrence suffixes when
case- or punctuation-distinct UCI fields collapse to the same snake-case name;
- keeps columns shared by all three sources;
- removes obvious identifiers and pre-existing target columns;
- converts usable columns to numeric values;
- adds `source` and `label` (`0 = benign`, `1 = malware`);
- retains missing values so imputation can be fitted on training data inside the notebooks;
- writes `data/malware_features.csv` and `data/feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
path = Path("data/malware_features.csv")
df = pd.read_csv(path)
print(df.shape)
print(df["label"].value_counts().sort_index())
print(df["source"].value_counts())
```
The original release contains approximately 6,248 rows: 595 benign records and 5,653 malware records. The exact number of retained feature columns can vary if the source archive changes column formatting.
## Provenance and license
- Dataset: Malware Static and Dynamic Features VxHeaven and Virus Total
- Source: UCI Machine Learning Repository
- DOI: [10.24432/C58K6H](https://doi.org/10.24432/C58K6H)
- License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
Retain the citation and license when redistributing derived classroom tables.
## Important limitations
- The class distribution is strongly imbalanced.
- Benign and malware samples originate from different collections, so source-specific artifacts may inflate random-split performance.
- The features represent a historical dataset and should not be treated as a current production benchmark.
- No raw malware is required or permitted for this practical.
Lab-5/Lab-5-Practical/data/feature_manifest.csv
+245
View File
@@ -0,0 +1,245 @@
feature,non_missing_ratio,unique_values,feature_family
add,1.0,2809,unclassified
addressofentrypoint,1.0,3156,unclassified
align,1.0,11,unclassified
arg,1.0,5,unclassified
asc,1.0,3,unclassified
baseofcode,1.0,145,unclassified
bool,1.0,2,unclassified
bt,1.0,114,unclassified
byte,1.0,17,unclassified
call,1.0,2131,unclassified
cexit,1.0,2,unclassified
char,1.0,4,unclassified
characteristics,1.0,52,unclassified
checksum,1.0,2100,unclassified
cld,1.0,685,unclassified
cli,1.0,625,unclassified
closehandle,1.0,2,unclassified
cmc,1.0,635,unclassified
cmp,1.0,1954,unclassified
compile_date,1.0,3912,unclassified
cookie,1.0,2,unclassified
copyfilea,1.0,2,unclassified
count_dll_loaded,1.0,60,unclassified
count_file_copied,1.0,47,unclassified
count_file_deleted,1.0,19,unclassified
count_file_exists,1.0,75,unclassified
count_file_opened,1.0,76,unclassified
count_file_read,1.0,29,unclassified
count_file_written,1.0,24,unclassified
count_mutex,1.0,18,unclassified
count_regkey_deleted,1.0,12,unclassified
count_regkey_written,1.0,45,unclassified
createdirectorya,1.0,2,unclassified
createfilea,1.0,2,unclassified
createmutexa,1.0,2,unclassified
createprocessa,1.0,2,unclassified
createthread,1.0,2,unclassified
createtoolhelp32snapshot,1.0,2,unclassified
cs,1.0,529,unclassified
daa,1.0,618,unclassified
data,1.0,4,unclassified
data__2,1.0,2,unclassified
db,1.0,1827,unclassified
db_1,1.0,1147,unclassified
dd,1.0,3111,unclassified
dd_1,1.0,946,unclassified
dec,1.0,1627,unclassified
deletefilea,1.0,2,unclassified
dll,1.0,225,unclassified
dll_characteristics,1.0,27,unclassified
ds,1.0,526,unclassified
dw,1.0,63,unclassified
enablewindow,1.0,2,unclassified
ent_max,1.0,5674,unclassified
ent_mean,1.0,6243,unclassified
ent_median,1.0,6087,unclassified
ent_min,1.0,5699,unclassified
ent_var,1.0,6114,unclassified
ent_whole_file,1.0,6248,unclassified
entry,1.0,5,unclassified
es,1.0,500,unclassified
exit,1.0,33,unclassified
exitprocess,1.0,3,unclassified
exitwindowsex,1.0,2,unclassified
faddp,1.0,131,unclassified
fchs,1.0,55,unclassified
fclose,1.0,2,unclassified
fdiv,1.0,175,unclassified
fdivp,1.0,130,unclassified
fdivr,1.0,192,unclassified
fild,1.0,276,unclassified
file_alignment,1.0,5,unclassified
files_operations,1.0,159,unclassified
filesize,1.0,5252,unclassified
findclose,1.0,2,unclassified
findfirstfilea,1.0,2,unclassified
findnextfilea,1.0,2,unclassified
findwindowa,1.0,2,unclassified
fistp,1.0,202,unclassified
fld,1.0,144,unclassified
fopen,1.0,3,unclassified
free,1.0,169,unclassified
fs,1.0,683,unclassified
fstcw,1.0,19,unclassified
fwrite,1.0,3,unclassified
fxch,1.0,148,unclassified
getcommandlinea,1.0,2,unclassified
getfileattributesa,1.0,2,unclassified
getforegroundwindow,1.0,2,unclassified
getlasterror,1.0,4,unclassified
getmainargs,1.0,2,unclassified
getmodulefilenamea,1.0,3,unclassified
getmodulehandlea,1.0,3,unclassified
getstartupinfoa,1.0,2,unclassified
getsystemdirectorya,1.0,2,unclassified
getsystemmetrics,1.0,2,unclassified
gettickcount,1.0,2,unclassified
gs,1.0,796,unclassified
handle,1.0,10,unclassified
image_base,1.0,44,unclassified
imul,1.0,1113,unclassified
in,1.0,1016,unclassified
inc,1.0,1941,unclassified
initterm,1.0,2,unclassified
int,1.0,1500,unclassified
int_1,1.0,575,unclassified
interlockedcompareexchange,1.0,2,unclassified
interlockeddecrement,1.0,2,unclassified
interlockedincrement,1.0,2,unclassified
jb,1.0,1089,unclassified
je,1.0,1732,unclassified
jg,1.0,765,unclassified
jge,1.0,748,unclassified
jl,1.0,825,unclassified
jmp,1.0,1707,unclassified
jno,1.0,650,unclassified
jo,1.0,819,unclassified
lea,1.0,1673,unclassified
loader_flags,1.0,4,unclassified
loc,1.0,688,unclassified
long,1.0,10,unclassified
loope,1.0,580,unclassified
lstrcata,1.0,2,unclassified
lstrcpya,1.0,2,unclassified
lstrlena,1.0,2,unclassified
machine,1.0,2,unclassified
magic,1.0,2,unclassified
major_image_version,1.0,39,unclassified
major_linker_version,1.0,30,unclassified
major_operating_system_version,1.0,7,unclassified
major_subsystem_version,1.0,5,unclassified
malloc,1.0,5,unclassified
max_min,1.0,5517,unclassified
memcpy,1.0,5,unclassified
memmove,1.0,3,unclassified
memset,1.0,5,unclassified
messageboxa,1.0,2,unclassified
minor_image_version,1.0,44,unclassified
minor_linker_version,1.0,47,unclassified
minor_operating_system_version,1.0,10,unclassified
minor_subsystem_version,1.0,6,unclassified
mov,1.0,3156,unclassified
mul,1.0,207,unclassified
neg,1.0,301,unclassified
nop,1.0,1037,unclassified
not,1.0,265,unclassified
number_of_iat_entires,1.0,467,unclassified
number_of_iat_entires_1,1.0,467,unclassified
number_of_imports_1,1.0,40,unclassified
number_of_rva_and_sizes,1.0,8,unclassified
number_of_rva_and_sizes_1,1.0,8,unclassified
number_of_sections_1,1.0,29,unclassified
number_of_symbols,1.0,50,unclassified
off,1.0,4,unclassified
or,1.0,1497,unclassified
out,1.0,1013,unclassified
p_fmode,1.0,2,unclassified
pointer_to_symbol_table,1.0,42,unclassified
pop,1.0,2306,unclassified
popf,1.0,585,unclassified
proc,1.0,2,unclassified
process32first,1.0,2,unclassified
process32next,1.0,2,unclassified
ptr,1.0,16,unclassified
push,1.0,3002,unclassified
pushf,1.0,642,unclassified
raiseexception,1.0,3,unclassified
rand,1.0,6,unclassified
rcl,1.0,314,unclassified
rcr,1.0,261,unclassified
rdata,1.0,2,unclassified
rdtsc,1.0,50,unclassified
regclosekey,1.0,3,unclassified
regopenkeya,1.0,2,unclassified
regopenkeyexa,1.0,2,unclassified
regsetvalueexa,1.0,3,unclassified
rep,1.0,263,unclassified
ret,1.0,1458,unclassified
rol,1.0,348,unclassified
ror,1.0,332,unclassified
sar,1.0,466,unclassified
sbb,1.0,1230,unclassified
scas,1.0,777,unclassified
section_alignment,1.0,5,unclassified
sendmessagea,1.0,2,unclassified
set_app_type,1.0,3,unclassified
setb,1.0,54,unclassified
setcursorpos,1.0,2,unclassified
setfileattributesa,1.0,3,unclassified
setlasterror,1.0,2,unclassified
setle,1.0,50,unclassified
setusermatherr,1.0,2,unclassified
setwindowtexta,1.0,2,unclassified
shellexecutea,1.0,2,unclassified
shl,1.0,493,unclassified
shld,1.0,79,unclassified
showwindow,1.0,2,unclassified
shr,1.0,499,unclassified
sidt,1.0,2,unclassified
size_code,1.0,930,unclassified
size_image,1.0,829,unclassified
size_init_data,1.0,948,unclassified
size_of_headers,1.0,30,unclassified
size_of_heap_commit,1.0,10,unclassified
size_of_heap_reserve,1.0,26,unclassified
size_of_optional_header,1.0,2,unclassified
size_of_stack_commit,1.0,35,unclassified
size_of_stack_reserve,1.0,34,unclassified
size_uninit_data,1.0,164,unclassified
sizeofheaders,1.0,30,unclassified
sizeofheaders_1,1.0,30,unclassified
sleep,1.0,4,unclassified
sprintf,1.0,2,unclassified
ss,1.0,577,unclassified
stc,1.0,634,unclassified
std,1.0,658,unclassified
sti,1.0,669,unclassified
stos,1.0,790,unclassified
strcat,1.0,4,unclassified
strchr,1.0,2,unclassified
strcmp,1.0,4,unclassified
strcpy,1.0,4,unclassified
strlen,1.0,6,unclassified
strncmp,1.0,2,unclassified
strstr,1.0,3,unclassified
sub,1.0,1631,unclassified
subsystem,1.0,3,unclassified
test,1.0,1685,unclassified
text,1.0,5,unclassified
time,1.0,5,unclassified
tlsalloc,1.0,2,unclassified
tlsfree,1.0,2,unclassified
tlsgetvalue,1.0,2,unclassified
tlssetvalue,1.0,2,unclassified
uint,1.0,3,unclassified
unhandledexceptionfilter,1.0,2,unclassified
waitforsingleobject,1.0,2,unclassified
winexec,1.0,2,unclassified
winmain,1.0,6,unclassified
word,1.0,9,unclassified
word__2,1.0,3,unclassified
writefile,1.0,2,unclassified
xchg,1.0,1217,unclassified
xor,1.0,1887,unclassified
1 feature non_missing_ratio unique_values feature_family
2 add 1.0 2809 unclassified
3 addressofentrypoint 1.0 3156 unclassified
4 align 1.0 11 unclassified
5 arg 1.0 5 unclassified
6 asc 1.0 3 unclassified
7 baseofcode 1.0 145 unclassified
8 bool 1.0 2 unclassified
9 bt 1.0 114 unclassified
10 byte 1.0 17 unclassified
11 call 1.0 2131 unclassified
12 cexit 1.0 2 unclassified
13 char 1.0 4 unclassified
14 characteristics 1.0 52 unclassified
15 checksum 1.0 2100 unclassified
16 cld 1.0 685 unclassified
17 cli 1.0 625 unclassified
18 closehandle 1.0 2 unclassified
19 cmc 1.0 635 unclassified
20 cmp 1.0 1954 unclassified
21 compile_date 1.0 3912 unclassified
22 cookie 1.0 2 unclassified
23 copyfilea 1.0 2 unclassified
24 count_dll_loaded 1.0 60 unclassified
25 count_file_copied 1.0 47 unclassified
26 count_file_deleted 1.0 19 unclassified
27 count_file_exists 1.0 75 unclassified
28 count_file_opened 1.0 76 unclassified
29 count_file_read 1.0 29 unclassified
30 count_file_written 1.0 24 unclassified
31 count_mutex 1.0 18 unclassified
32 count_regkey_deleted 1.0 12 unclassified
33 count_regkey_written 1.0 45 unclassified
34 createdirectorya 1.0 2 unclassified
35 createfilea 1.0 2 unclassified
36 createmutexa 1.0 2 unclassified
37 createprocessa 1.0 2 unclassified
38 createthread 1.0 2 unclassified
39 createtoolhelp32snapshot 1.0 2 unclassified
40 cs 1.0 529 unclassified
41 daa 1.0 618 unclassified
42 data 1.0 4 unclassified
43 data__2 1.0 2 unclassified
44 db 1.0 1827 unclassified
45 db_1 1.0 1147 unclassified
46 dd 1.0 3111 unclassified
47 dd_1 1.0 946 unclassified
48 dec 1.0 1627 unclassified
49 deletefilea 1.0 2 unclassified
50 dll 1.0 225 unclassified
51 dll_characteristics 1.0 27 unclassified
52 ds 1.0 526 unclassified
53 dw 1.0 63 unclassified
54 enablewindow 1.0 2 unclassified
55 ent_max 1.0 5674 unclassified
56 ent_mean 1.0 6243 unclassified
57 ent_median 1.0 6087 unclassified
58 ent_min 1.0 5699 unclassified
59 ent_var 1.0 6114 unclassified
60 ent_whole_file 1.0 6248 unclassified
61 entry 1.0 5 unclassified
62 es 1.0 500 unclassified
63 exit 1.0 33 unclassified
64 exitprocess 1.0 3 unclassified
65 exitwindowsex 1.0 2 unclassified
66 faddp 1.0 131 unclassified
67 fchs 1.0 55 unclassified
68 fclose 1.0 2 unclassified
69 fdiv 1.0 175 unclassified
70 fdivp 1.0 130 unclassified
71 fdivr 1.0 192 unclassified
72 fild 1.0 276 unclassified
73 file_alignment 1.0 5 unclassified
74 files_operations 1.0 159 unclassified
75 filesize 1.0 5252 unclassified
76 findclose 1.0 2 unclassified
77 findfirstfilea 1.0 2 unclassified
78 findnextfilea 1.0 2 unclassified
79 findwindowa 1.0 2 unclassified
80 fistp 1.0 202 unclassified
81 fld 1.0 144 unclassified
82 fopen 1.0 3 unclassified
83 free 1.0 169 unclassified
84 fs 1.0 683 unclassified
85 fstcw 1.0 19 unclassified
86 fwrite 1.0 3 unclassified
87 fxch 1.0 148 unclassified
88 getcommandlinea 1.0 2 unclassified
89 getfileattributesa 1.0 2 unclassified
90 getforegroundwindow 1.0 2 unclassified
91 getlasterror 1.0 4 unclassified
92 getmainargs 1.0 2 unclassified
93 getmodulefilenamea 1.0 3 unclassified
94 getmodulehandlea 1.0 3 unclassified
95 getstartupinfoa 1.0 2 unclassified
96 getsystemdirectorya 1.0 2 unclassified
97 getsystemmetrics 1.0 2 unclassified
98 gettickcount 1.0 2 unclassified
99 gs 1.0 796 unclassified
100 handle 1.0 10 unclassified
101 image_base 1.0 44 unclassified
102 imul 1.0 1113 unclassified
103 in 1.0 1016 unclassified
104 inc 1.0 1941 unclassified
105 initterm 1.0 2 unclassified
106 int 1.0 1500 unclassified
107 int_1 1.0 575 unclassified
108 interlockedcompareexchange 1.0 2 unclassified
109 interlockeddecrement 1.0 2 unclassified
110 interlockedincrement 1.0 2 unclassified
111 jb 1.0 1089 unclassified
112 je 1.0 1732 unclassified
113 jg 1.0 765 unclassified
114 jge 1.0 748 unclassified
115 jl 1.0 825 unclassified
116 jmp 1.0 1707 unclassified
117 jno 1.0 650 unclassified
118 jo 1.0 819 unclassified
119 lea 1.0 1673 unclassified
120 loader_flags 1.0 4 unclassified
121 loc 1.0 688 unclassified
122 long 1.0 10 unclassified
123 loope 1.0 580 unclassified
124 lstrcata 1.0 2 unclassified
125 lstrcpya 1.0 2 unclassified
126 lstrlena 1.0 2 unclassified
127 machine 1.0 2 unclassified
128 magic 1.0 2 unclassified
129 major_image_version 1.0 39 unclassified
130 major_linker_version 1.0 30 unclassified
131 major_operating_system_version 1.0 7 unclassified
132 major_subsystem_version 1.0 5 unclassified
133 malloc 1.0 5 unclassified
134 max_min 1.0 5517 unclassified
135 memcpy 1.0 5 unclassified
136 memmove 1.0 3 unclassified
137 memset 1.0 5 unclassified
138 messageboxa 1.0 2 unclassified
139 minor_image_version 1.0 44 unclassified
140 minor_linker_version 1.0 47 unclassified
141 minor_operating_system_version 1.0 10 unclassified
142 minor_subsystem_version 1.0 6 unclassified
143 mov 1.0 3156 unclassified
144 mul 1.0 207 unclassified
145 neg 1.0 301 unclassified
146 nop 1.0 1037 unclassified
147 not 1.0 265 unclassified
148 number_of_iat_entires 1.0 467 unclassified
149 number_of_iat_entires_1 1.0 467 unclassified
150 number_of_imports_1 1.0 40 unclassified
151 number_of_rva_and_sizes 1.0 8 unclassified
152 number_of_rva_and_sizes_1 1.0 8 unclassified
153 number_of_sections_1 1.0 29 unclassified
154 number_of_symbols 1.0 50 unclassified
155 off 1.0 4 unclassified
156 or 1.0 1497 unclassified
157 out 1.0 1013 unclassified
158 p_fmode 1.0 2 unclassified
159 pointer_to_symbol_table 1.0 42 unclassified
160 pop 1.0 2306 unclassified
161 popf 1.0 585 unclassified
162 proc 1.0 2 unclassified
163 process32first 1.0 2 unclassified
164 process32next 1.0 2 unclassified
165 ptr 1.0 16 unclassified
166 push 1.0 3002 unclassified
167 pushf 1.0 642 unclassified
168 raiseexception 1.0 3 unclassified
169 rand 1.0 6 unclassified
170 rcl 1.0 314 unclassified
171 rcr 1.0 261 unclassified
172 rdata 1.0 2 unclassified
173 rdtsc 1.0 50 unclassified
174 regclosekey 1.0 3 unclassified
175 regopenkeya 1.0 2 unclassified
176 regopenkeyexa 1.0 2 unclassified
177 regsetvalueexa 1.0 3 unclassified
178 rep 1.0 263 unclassified
179 ret 1.0 1458 unclassified
180 rol 1.0 348 unclassified
181 ror 1.0 332 unclassified
182 sar 1.0 466 unclassified
183 sbb 1.0 1230 unclassified
184 scas 1.0 777 unclassified
185 section_alignment 1.0 5 unclassified
186 sendmessagea 1.0 2 unclassified
187 set_app_type 1.0 3 unclassified
188 setb 1.0 54 unclassified
189 setcursorpos 1.0 2 unclassified
190 setfileattributesa 1.0 3 unclassified
191 setlasterror 1.0 2 unclassified
192 setle 1.0 50 unclassified
193 setusermatherr 1.0 2 unclassified
194 setwindowtexta 1.0 2 unclassified
195 shellexecutea 1.0 2 unclassified
196 shl 1.0 493 unclassified
197 shld 1.0 79 unclassified
198 showwindow 1.0 2 unclassified
199 shr 1.0 499 unclassified
200 sidt 1.0 2 unclassified
201 size_code 1.0 930 unclassified
202 size_image 1.0 829 unclassified
203 size_init_data 1.0 948 unclassified
204 size_of_headers 1.0 30 unclassified
205 size_of_heap_commit 1.0 10 unclassified
206 size_of_heap_reserve 1.0 26 unclassified
207 size_of_optional_header 1.0 2 unclassified
208 size_of_stack_commit 1.0 35 unclassified
209 size_of_stack_reserve 1.0 34 unclassified
210 size_uninit_data 1.0 164 unclassified
211 sizeofheaders 1.0 30 unclassified
212 sizeofheaders_1 1.0 30 unclassified
213 sleep 1.0 4 unclassified
214 sprintf 1.0 2 unclassified
215 ss 1.0 577 unclassified
216 stc 1.0 634 unclassified
217 std 1.0 658 unclassified
218 sti 1.0 669 unclassified
219 stos 1.0 790 unclassified
220 strcat 1.0 4 unclassified
221 strchr 1.0 2 unclassified
222 strcmp 1.0 4 unclassified
223 strcpy 1.0 4 unclassified
224 strlen 1.0 6 unclassified
225 strncmp 1.0 2 unclassified
226 strstr 1.0 3 unclassified
227 sub 1.0 1631 unclassified
228 subsystem 1.0 3 unclassified
229 test 1.0 1685 unclassified
230 text 1.0 5 unclassified
231 time 1.0 5 unclassified
232 tlsalloc 1.0 2 unclassified
233 tlsfree 1.0 2 unclassified
234 tlsgetvalue 1.0 2 unclassified
235 tlssetvalue 1.0 2 unclassified
236 uint 1.0 3 unclassified
237 unhandledexceptionfilter 1.0 2 unclassified
238 waitforsingleobject 1.0 2 unclassified
239 winexec 1.0 2 unclassified
240 winmain 1.0 6 unclassified
241 word 1.0 9 unclassified
242 word__2 1.0 3 unclassified
243 writefile 1.0 2 unclassified
244 xchg 1.0 1217 unclassified
245 xor 1.0 1887 unclassified
Lab-5/Lab-5-Practical/data/malware_features.csv
+6249
View File
File diff suppressed because it is too large Load Diff
Lab-5/Lab-5-Practical/data/prepare_dataset.py
+168
View File
@@ -0,0 +1,168 @@
"""Prepare a safe classroom feature table from the three UCI CSV files."""
from pathlib import Path
import re
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
OUTPUT_PATH = HERE / "malware_features.csv"
MANIFEST_PATH = HERE / "feature_manifest.csv"
FILES = {
"benign_lab": ("staDynBenignLab.csv", 0),
"vxheaven": ("staDynVxHeaven2698Lab.csv", 1),
"virustotal": ("staDynVt2955Lab.csv", 1),
}
FORBIDDEN_NAMES = {
"label", "class", "target", "malware", "is_malware", "type",
"file", "filename", "file_name", "name", "hash", "md5", "sha1",
"sha256", "source", "category",
}
def normalize_name(value: object) -> str:
"""Convert a source column name to a stable snake_case identifier."""
text = re.sub(r"[^0-9A-Za-z]+", "_", str(value).strip()).strip("_").lower()
return text or "unnamed_feature"
def normalize_columns(columns: list[object]) -> tuple[list[str], dict[str, list[str]]]:
"""Normalize names and disambiguate collisions by their source-column order.
The UCI files intentionally contain case- or punctuation-distinct fields such as
``WORD``/``word`` and ``Import``/``__IMPORT_``. A simple lowercase snake-case
conversion collapses each pair. The three source files use the same relative
order for these fields, so an occurrence suffix produces stable shared names:
``word``, ``word__2``, and so on.
"""
counts: dict[str, int] = {}
normalized: list[str] = []
originals_by_base: dict[str, list[str]] = {}
for raw_column in columns:
base = normalize_name(raw_column)
counts[base] = counts.get(base, 0) + 1
occurrence = counts[base]
unique_name = base if occurrence == 1 else f"{base}__{occurrence}"
normalized.append(unique_name)
originals_by_base.setdefault(base, []).append(str(raw_column))
collisions = {
base: originals
for base, originals in originals_by_base.items()
if len(originals) > 1
}
if len(normalized) != len(set(normalized)):
raise ValueError("Column names are still duplicated after disambiguation.")
return normalized, collisions
def find_required_file(filename: str) -> Path:
"""Find one required CSV below data/raw, allowing an extracted subfolder."""
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and check the extracted folder structure."
)
return matches[0]
def load_source(source: str, filename: str, label: int) -> pd.DataFrame:
"""Load one source file and attach provenance columns."""
frame = pd.read_csv(find_required_file(filename), low_memory=False)
normalized, collisions = normalize_columns(frame.columns.tolist())
frame.columns = normalized
# Consolidate the wide table before adding provenance columns. This avoids
# pandas fragmentation warnings on the 1,000+ column UCI source files.
frame = frame.copy()
frame.attrs["normalized_column_collisions"] = collisions
frame["source"] = source
frame["label"] = label
return frame
def validate_collision_alignment(frames: dict[str, pd.DataFrame]) -> None:
"""Ensure occurrence suffixes refer to the same raw fields in every source."""
collision_names = set().union(*(
set(frame.attrs.get("normalized_column_collisions", {}))
for frame in frames.values()
))
for base in sorted(collision_names):
variants = {
source: tuple(frame.attrs.get("normalized_column_collisions", {}).get(base, []))
for source, frame in frames.items()
}
if len(set(variants.values())) != 1:
raise ValueError(
f"Normalized collision {base!r} is not aligned across source files: {variants}"
)
if collision_names:
print("Disambiguated normalized column collisions:")
reference = next(iter(frames.values())).attrs["normalized_column_collisions"]
for base in sorted(collision_names):
generated = [base] + [f"{base}__{index}" for index in range(2, len(reference[base]) + 1)]
print(f" {reference[base]} -> {generated}")
def main() -> None:
frames = {
source: load_source(source, filename, label)
for source, (filename, label) in FILES.items()
}
validate_collision_alignment(frames)
common = set.intersection(*(set(frame.columns) for frame in frames.values()))
candidate_features = sorted(common - FORBIDDEN_NAMES - {"source", "label"})
if not candidate_features:
raise ValueError("No shared candidate feature columns were found.")
combined_parts = []
for source, frame in frames.items():
part = frame[candidate_features].copy()
part.insert(0, "source", source)
part.insert(1, "label", int(FILES[source][1]))
combined_parts.append(part)
combined = pd.concat(combined_parts, ignore_index=True)
numeric = combined[candidate_features].apply(pd.to_numeric, errors="coerce")
numeric = numeric.replace([np.inf, -np.inf], np.nan)
usable_ratio = numeric.notna().mean()
retained = usable_ratio[usable_ratio >= 0.95].index.tolist()
numeric = numeric[retained]
non_constant = numeric.nunique(dropna=True)
retained = non_constant[non_constant > 1].index.tolist()
numeric = numeric[retained]
if numeric.empty:
raise ValueError("No usable numeric features remained after validation.")
prepared = pd.concat([combined[["source", "label"]], numeric], axis=1)
prepared = prepared.sample(frac=1.0, random_state=42).reset_index(drop=True)
prepared.to_csv(OUTPUT_PATH, index=False)
manifest = pd.DataFrame({
"feature": retained,
"non_missing_ratio": numeric[retained].notna().mean().values,
"unique_values": numeric[retained].nunique(dropna=True).values,
"feature_family": "unclassified",
})
manifest.to_csv(MANIFEST_PATH, index=False)
print(f"Created {OUTPUT_PATH}")
print(f"Shape: {prepared.shape}")
print("Class counts:")
print(prepared["label"].value_counts().sort_index())
print("Source counts:")
print(prepared["source"].value_counts())
print(f"Created {MANIFEST_PATH}")
if __name__ == "__main__":
main()
Lab-5/Lab-5-Practical/data/raw/staDynBenignLab.csv
+596
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Practical/data/raw/staDynVt2955Lab.csv
+2956
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Practical/data/raw/staDynVxHeaven2698Lab.csv
+2699
View File
File diff suppressed because one or more lines are too long