labs/Lab-3/Lab-3-Solution/06_Capstone_CICIDS2017_Binary_IDS_Solution.ipynb

Module 3 — Lab 6: Capstone — Binary Intrusion Detection with CIC-IDS2017

Topics: end-to-end ML workflow, CIC-IDS2017, binary classification, evaluation, reporting

Solution version: This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.

Mission

Build one complete and leakage-safe intrusion-detection workflow:

1. load a network-flow table;
2. clean invalid values and duplicate rows;
3. map all attacks to one ATTACK class;
4. create one stratified train-test split;
5. train one Random Forest pipeline;
6. evaluate attack precision, recall, F1, and error counts;
7. export a short analyst report.

The notebook uses data/cicids2017_binary_sample.csv when that file is available. Until the instructor prepares the official subset, it uses the included synthetic dataset and prints a clear warning.

In [1]:

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    ConfusionMatrixDisplay, accuracy_score, confusion_matrix,
    precision_score, recall_score, f1_score
)

REAL_DATA_PATH = Path("data/cicids2017_binary_sample.csv")
DEMO_DATA_PATH = Path("data/synthetic_security_flows.csv")
OUTPUT_DIR = Path("output")
OUTPUT_DIR.mkdir(exist_ok=True)
RANDOM_STATE = 42

DATA_PATH = REAL_DATA_PATH if REAL_DATA_PATH.exists() else DEMO_DATA_PATH
USING_REAL_CICIDS = REAL_DATA_PATH.exists()

print("Dataset:", DATA_PATH)
if not USING_REAL_CICIDS:
    print("WARNING: Using synthetic classroom data. Add the prepared CIC-IDS2017 file for the real capstone.")

Dataset: data/cicids2017_binary_sample.csv

1. Load, clean, and define the binary target

In [2]:

raw_df = pd.read_csv(DATA_PATH, low_memory=False)
raw_df.columns = raw_df.columns.str.strip()
clean_df = raw_df.drop_duplicates().replace([np.inf, -np.inf], np.nan)

if "Label" not in clean_df.columns:
    raise KeyError("Expected a column named 'Label'. Check data/README.md.")

X = clean_df.drop(columns="Label").select_dtypes(include="number")
y = clean_df["Label"].astype(str).str.strip().str.upper().ne("BENIGN").astype(int)

print(f"Raw rows: {len(raw_df):,}")
print(f"Clean rows: {len(clean_df):,}")
print(f"Numeric features: {X.shape[1]}")
print("Binary class distribution:")
print(y.value_counts().sort_index().rename(index={0: "BENIGN", 1: "ATTACK"}))

Raw rows: 50,000
Clean rows: 50,000
Numeric features: 78
Binary class distribution:
Label
BENIGN    40000
ATTACK    10000
Name: count, dtype: int64

2. Create one fixed train-test split

In [3]:

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=RANDOM_STATE,
    stratify=y,
)

print("Training samples:", len(X_train))
print("Test samples:", len(X_test))
print("Training attack rate:", round(y_train.mean(), 3))
print("Test attack rate:", round(y_test.mean(), 3))

Training samples: 40000
Test samples: 10000
Training attack rate: 0.2
Test attack rate: 0.2

3. Build and train the baseline pipeline

In [4]:

baseline_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("model", RandomForestClassifier(
        n_estimators=150,
        max_depth=14,
        random_state=RANDOM_STATE,
        n_jobs=-1,
    )),
])

baseline_pipeline.fit(X_train, y_train)
y_pred = baseline_pipeline.predict(X_test)

4. Evaluate the attack class

In [5]:

cm = confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = cm.ravel()

baseline_metrics = {
    "accuracy": accuracy_score(y_test, y_pred),
    "attack_precision": precision_score(y_test, y_pred, zero_division=0),
    "attack_recall": recall_score(y_test, y_pred, zero_division=0),
    "attack_f1": f1_score(y_test, y_pred, zero_division=0),
    "false_positives": int(fp),
    "false_negatives": int(fn),
}

print(pd.Series(baseline_metrics).round(3))
print(f"False positives: {fp} | False negatives: {fn}")

display = ConfusionMatrixDisplay(cm, display_labels=["BENIGN", "ATTACK"])
display.plot(cmap="Blues")
plt.title("CIC-IDS2017 binary intrusion detector")
plt.tight_layout()
plt.savefig(OUTPUT_DIR / "capstone_confusion_matrix.png", dpi=150, bbox_inches="tight")
plt.show()

accuracy            1.000
attack_precision    1.000
attack_recall       0.998
attack_f1           0.999
false_positives     0.000
false_negatives     5.000
dtype: float64
False positives: 0 | False negatives: 5

5. Inspect feature importance

In [6]:

feature_importance = pd.Series(
    baseline_pipeline.named_steps["model"].feature_importances_,
    index=X.columns,
    name="importance",
).sort_values(ascending=False)

print(feature_importance.head(10))

ax = feature_importance.head(10).sort_values().plot(
    kind="barh",
    figsize=(8, 5),
    color="#2a9d8f",
    title="Top feature importance",
)
ax.set_xlabel("Random Forest importance")
ax.set_ylabel("Feature")
plt.tight_layout()
plt.savefig(OUTPUT_DIR / "capstone_feature_importance.png", dpi=150, bbox_inches="tight")
plt.show()

Bwd Packet Length Max          0.087949
Packet Length Std              0.078237
Bwd Packet Length Std          0.076631
Avg Bwd Segment Size           0.075204
Packet Length Variance         0.071238
Max Packet Length              0.059495
Bwd Packet Length Mean         0.045845
Total Length of Bwd Packets    0.039864
Average Packet Size            0.038397
Packet Length Mean             0.033550
Name: importance, dtype: float64

6. Export metrics and write an analyst summary

In [7]:

pd.Series(baseline_metrics, name="value").to_csv(
    OUTPUT_DIR / "capstone_metrics.csv",
    header=True,
)

dataset_type = "official CIC-IDS2017 subset" if USING_REAL_CICIDS else "synthetic classroom dataset"
summary = f'''BINARY INTRUSION DETECTION SUMMARY
==================================
Dataset: {dataset_type}
Clean samples: {len(clean_df):,}
Numeric features: {X.shape[1]}
Attack rate: {y.mean():.2%}
Attack precision: {baseline_metrics['attack_precision']:.3f}
Attack recall: {baseline_metrics['attack_recall']:.3f}
Attack F1: {baseline_metrics['attack_f1']:.3f}
False positives: {fp}
False negatives: {fn}

Recommendation:
- Review false negatives because they represent missed attacks.
- Review false positives before using this model for automated blocking.
- Validate the workflow on a time-based or group-based split before deployment.
'''

(OUTPUT_DIR / "capstone_summary.txt").write_text(summary, encoding="utf-8")
print(summary)

BINARY INTRUSION DETECTION SUMMARY
==================================
Dataset: official CIC-IDS2017 subset
Clean samples: 50,000
Numeric features: 78
Attack rate: 20.00%
Attack precision: 1.000
Attack recall: 0.998
Attack F1: 0.999
False positives: 0
False negatives: 5

Recommendation:
- Review false negatives because they represent missed attacks.
- Review false positives before using this model for automated blocking.
- Validate the workflow on a time-based or group-based split before deployment.

Self-check

In [8]:

assert isinstance(clean_df, pd.DataFrame)
assert set(y.unique()) == {0, 1}
assert X.shape[1] >= 5
assert abs(y_train.mean() - y_test.mean()) < 0.02
assert cm.shape == (2, 2)
assert len(feature_importance) == X.shape[1]
assert (OUTPUT_DIR / "capstone_confusion_matrix.png").exists()
assert (OUTPUT_DIR / "capstone_feature_importance.png").exists()
assert (OUTPUT_DIR / "capstone_metrics.csv").exists()
assert (OUTPUT_DIR / "capstone_summary.txt").exists()
print("✓ Capstone lab completed")

✓ Capstone lab completed

Capstone questions

1. How many attacks did the model miss, and why does that number matter operationally?
2. Which features received the highest importance? Do they make sense for network flows?
3. Why does random row splitting provide an optimistic estimate for some security datasets?
4. What additional validation would you require before deploying this detector?
5. If the notebook used synthetic data, which conclusions must be rechecked on the official CIC-IDS2017 subset?