labs/Lab-6/Lab-6-Solution/02_Flow_Features_Preprocessing_Solution.ipynb

Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing

Topics: feature families, log transform, encoding, scaling, train-only fitting

Solution version: Run the cells in order, compare the outputs with your work, and read the suggested answers.

Learning objectives

After completing this lab, you will be able to:

• interpret basic, content, traffic, and host-based flow features;
• apply log1p to skewed byte counts without deleting zero-byte flows;
• compare normal and attack distributions;
• build separate preprocessing plans for distance models and tree models;
• fit encoders and scalers using training data only.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

TRAIN_PATH = Path("data/nsl_kdd_train.csv")
TEST_PATH = Path("data/nsl_kdd_test.csv")

missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]
if missing_files:
    raise FileNotFoundError(
        "Missing prepared NSL-KDD files: " + ", ".join(missing_files) +
        ". Follow data/README.md and run: python data/prepare_dataset.py"
    )

TARGET_COLUMNS = ["attack_label", "attack_group", "binary_label", "difficulty"]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]


from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

In [ ]:

train_df = pd.read_csv(TRAIN_PATH, low_memory=False)
test_df = pd.read_csv(TEST_PATH, low_memory=False)
feature_cols = [column for column in train_df.columns if column not in TARGET_COLUMNS]
numeric_features = [column for column in feature_cols if column not in CATEGORICAL_FEATURES]

feature_families = {
    "basic": ["duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes", "land", "wrong_fragment", "urgent"],
    "content": numeric_features[6:19],
    "traffic": numeric_features[19:28],
    "host_based": numeric_features[28:],
}
{family: len(names) for family, names in feature_families.items()}

1. Inspect byte-count skew and apply log1p

In [ ]:

exploratory = train_df[["src_bytes", "dst_bytes", "binary_label"]].copy()
exploratory["log_src_bytes"] = np.log1p(exploratory["src_bytes"])
exploratory["log_dst_bytes"] = np.log1p(exploratory["dst_bytes"])

byte_summary = exploratory.groupby("binary_label")[[
    "src_bytes", "dst_bytes", "log_src_bytes", "log_dst_bytes"
]].median()
display(byte_summary)

sample = exploratory.sample(min(5000, len(exploratory)), random_state=42)
colors = sample["binary_label"].map({0: "#4C78A8", 1: "#E45756"})
ax = sample.plot.scatter(
    x="log_src_bytes", y="log_dst_bytes", c=colors, alpha=0.25, figsize=(8, 6)
)
ax.set_title("NSL-KDD flows after log1p byte transformation")
plt.tight_layout()
plt.show()

2. Build preprocessing plans for two model families

In [ ]:

supervised_preprocessor = ColumnTransformer([
    ("categorical", OneHotEncoder(handle_unknown="ignore"), CATEGORICAL_FEATURES),
    ("numeric", SimpleImputer(strategy="median"), numeric_features),
])

distance_features = [
    "duration", "src_bytes", "dst_bytes", "count", "srv_count",
    "serror_rate", "rerror_rate", "dst_host_count", "dst_host_srv_count",
]
distance_preprocessor = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

supervised_preprocessor.fit(train_df[feature_cols])
distance_preprocessor.fit(train_df[distance_features])
supervised_shape = supervised_preprocessor.transform(train_df[feature_cols].head(100)).shape
distance_shape = distance_preprocessor.transform(train_df[distance_features].head(100)).shape
print("Supervised transformed shape:", supervised_shape)
print("Distance transformed shape:", distance_shape)

Why two plans?

Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy.

Self-check

In [ ]:

assert len(feature_cols) == 41
assert exploratory[["log_src_bytes", "log_dst_bytes"]].ge(0).all().all()
assert supervised_shape[0] == 100
assert supervised_shape[1] > len(feature_cols)
assert distance_shape == (100, len(distance_features))
assert all(feature in numeric_features for feature in distance_features)
print("✓ Lab 2 completed")

Suggested answers

1. Why log1p? It compresses large byte values and remains defined at zero.
2. Why scale distance features? Otherwise byte counts can dominate rate features in Euclidean distance.
3. Why fit on training only? Test categories and distribution statistics must not influence the learned preprocessing state.