labs/Lab-5/Lab-5-Practical/02_Malware_Feature_Exploration_Lab.ipynb

Module 5 — Lab 2: Exploring Malware Feature Representations

Topics: feature families, distributions, effect size, responsible EDA

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• summarize numeric malware features by class;
• identify interpretable feature-name families;
• compare benign and malware distributions with robust statistics;
• distinguish exploratory association from causal evidence;
• avoid using the held-out test set for model-driven feature selection.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )

In [ ]:

# TODO 1: load the prepared dataset and define X, y, and feature_cols.
df = None
feature_cols = None
X = None
y = None
print("Rows:", None, "Features:", None)

1. Inspect feature-name families

Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy.

In [ ]:

family_tokens = {
    "pe_or_header": ("header", "section", "image", "size", "character", "linker", "dll", "entry"),
    "api_or_import": ("api", "import", "call", "function"),
    "byte_or_opcode": ("byte", "opcode", "asm", "hex"),
    "dynamic_behavior": ("registry", "network", "process", "file", "mutex", "service"),
}

# TODO 2: collect up to ten feature-name examples for each family.
family_examples = None
family_examples

2. Build a robust class-comparison profile

Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only.

In [ ]:

benign = X.loc[y == 0]
malware = X.loc[y == 1]

# TODO 3: calculate the IQR for every feature and replace zero IQR values with NaN.
iqr = None

# TODO 4: build feature_profile with missing rate, unique count, and class medians.
feature_profile = None

# TODO 5: calculate the absolute median gap divided by IQR, then select the top eight names.
top_features = None
display(feature_profile.loc[top_features])

3. Visualize class medians for the strongest exploratory differences

In [ ]:

# TODO 6: create a horizontal grouped bar chart of benign and malware medians.
median_plot = None
ax = None
plt.tight_layout()
plt.show()

Interpretation warning

A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset.

Self-check

In [ ]:

assert set(X.columns) == set(feature_cols)
assert set(y.unique()) == {0, 1}
assert len(feature_profile) == len(feature_cols)
assert len(top_features) == 8
assert set(top_features).issubset(feature_cols)
assert {"benign_median", "malware_median", "standardized_median_gap"}.issubset(feature_profile.columns)
print("✓ Lab 2 completed")

Reflection questions

1. Why are medians and IQR useful for security features?
2. Can a large standardized median gap prove that a feature causes malware behavior?
3. Why should this full-dataset ranking not be used directly to optimize the final model?