labs/Lab-5/Lab-5-Solution/02_Malware_Feature_Exploration_Solution.ipynb

Module 5 — Lab 2: Exploring Malware Feature Representations

Topics: feature families, distributions, effect size, responsible EDA

Solution version: Run the cells in order, compare the outputs with your work, and read the suggested answers.

Learning objectives

After completing this lab, you will be able to:

• summarize numeric malware features by class;
• identify interpretable feature-name families;
• compare benign and malware distributions with robust statistics;
• distinguish exploratory association from causal evidence;
• avoid using the held-out test set for model-driven feature selection.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

DATA_PATH = Path("data/malware_features.csv")

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/malware_features.csv. Follow data/README.md, place the three "
        "official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py"
    )

In [ ]:

df = pd.read_csv(DATA_PATH, low_memory=False)
feature_cols = [column for column in df.columns if column not in {"source", "label"}]
X = df[feature_cols].apply(pd.to_numeric, errors="coerce")
y = df["label"].astype(int)
print("Rows:", len(df), "Features:", len(feature_cols))

1. Inspect feature-name families

Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy.

In [ ]:

family_tokens = {
    "pe_or_header": ("header", "section", "image", "size", "character", "linker", "dll", "entry"),
    "api_or_import": ("api", "import", "call", "function"),
    "byte_or_opcode": ("byte", "opcode", "asm", "hex"),
    "dynamic_behavior": ("registry", "network", "process", "file", "mutex", "service"),
}

family_examples = {}
for family, tokens in family_tokens.items():
    matches = [name for name in feature_cols if any(token in name.lower() for token in tokens)]
    family_examples[family] = matches[:10]
family_examples

2. Build a robust class-comparison profile

Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only.

In [ ]:

benign = X.loc[y == 0]
malware = X.loc[y == 1]
q75 = X.quantile(0.75)
q25 = X.quantile(0.25)
iqr = (q75 - q25).replace(0, np.nan)

feature_profile = pd.DataFrame({
    "missing_rate": X.isna().mean(),
    "unique_values": X.nunique(dropna=True),
    "benign_median": benign.median(),
    "malware_median": malware.median(),
})
feature_profile["standardized_median_gap"] = (
    (feature_profile["malware_median"] - feature_profile["benign_median"]).abs() / iqr
)
feature_profile = feature_profile.replace([np.inf, -np.inf], np.nan)
top_features = feature_profile["standardized_median_gap"].nlargest(8).index.tolist()
display(feature_profile.loc[top_features])

3. Visualize class medians for the strongest exploratory differences

In [ ]:

median_plot = feature_profile.loc[top_features, ["benign_median", "malware_median"]]
ax = median_plot.plot(kind="barh", figsize=(10, 6))
ax.set_title("Class medians for exploratory high-gap features")
ax.set_xlabel("Median feature value")
ax.set_ylabel("Feature")
plt.tight_layout()
plt.show()

Interpretation warning

A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset.

Self-check

In [ ]:

assert set(X.columns) == set(feature_cols)
assert set(y.unique()) == {0, 1}
assert len(feature_profile) == len(feature_cols)
assert len(top_features) == 8
assert set(top_features).issubset(feature_cols)
assert {"benign_median", "malware_median", "standardized_median_gap"}.issubset(feature_profile.columns)
print("✓ Lab 2 completed")

Suggested answers

1. Why use medians and IQR? They are less sensitive to extreme values and skew than means and standard deviations.
2. Can the ranking prove causation? No. It identifies associations in this particular collection.
3. Why not choose final features from the full table? Doing so lets test-set patterns influence model design and makes the final score optimistic.