labs/Lab-6/Lab-6-Practical/02_Flow_Features_Preprocessing_Lab.ipynb

Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing

Topics: feature families, log transform, encoding, scaling, train-only fitting

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• interpret basic, content, traffic, and host-based flow features;
• apply log1p to skewed byte counts without deleting zero-byte flows;
• compare normal and attack distributions;
• build separate preprocessing plans for distance models and tree models;
• fit encoders and scalers using training data only.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

TRAIN_PATH = Path("data/nsl_kdd_train.csv")
TEST_PATH = Path("data/nsl_kdd_test.csv")

missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]
if missing_files:
    raise FileNotFoundError(
        "Missing prepared NSL-KDD files: " + ", ".join(missing_files) +
        ". Follow data/README.md and run: python data/prepare_dataset.py"
    )

TARGET_COLUMNS = ["attack_label", "attack_group", "binary_label", "difficulty"]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]


from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

In [ ]:

# TODO 1: load both splits and define feature_cols/numeric_features.
train_df = None
test_df = None
feature_cols = None
numeric_features = None

# TODO 2: create feature_families for basic, content, traffic, and host-based features.
feature_families = None

1. Inspect byte-count skew and apply log1p

In [ ]:

# TODO 3: create log_src_bytes and log_dst_bytes using np.log1p.
exploratory = None

# TODO 4: calculate class medians and plot a sampled log-byte scatter chart.
byte_summary = None
sample = None
display(byte_summary)

2. Build preprocessing plans for two model families

In [ ]:

# TODO 5: build supervised_preprocessor with OneHotEncoder for the three
# categorical features and median imputation for numeric features.
supervised_preprocessor = None

distance_features = [
    "duration", "src_bytes", "dst_bytes", "count", "srv_count",
    "serror_rate", "rerror_rate", "dst_host_count", "dst_host_srv_count",
]
# TODO 6: build distance_preprocessor with median imputation and StandardScaler.
distance_preprocessor = None

# TODO 7: fit both preprocessors on training data only and record output shapes.
supervised_shape = None
distance_shape = None
print(supervised_shape, distance_shape)

Why two plans?

Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy.

Self-check

In [ ]:

assert len(feature_cols) == 41
assert exploratory[["log_src_bytes", "log_dst_bytes"]].ge(0).all().all()
assert supervised_shape[0] == 100
assert supervised_shape[1] > len(feature_cols)
assert distance_shape == (100, len(distance_features))
assert all(feature in numeric_features for feature in distance_features)
print("✓ Lab 2 completed")

Reflection questions

1. Why is log1p preferable to dropping zero-byte flows?
2. Why do distance methods need scaling while Decision Trees usually do not?
3. What information leaks when a scaler or encoder is fitted on test data?