labs/Lab-6/Lab-6-Practical/data/README.md

Dataset setup — NSL-KDD

All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.

Download

The official UNB NSL-KDD page documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.

Download these two files from the NSL-KDD GitHub mirror:

KDDTrain+_20Percent.txt
KDDTest+.txt

A preserved copy is also available through the NSL-KDD Zenodo record. If the Zenodo record does not include the 20% training file, download KDDTrain+.txt; the preparation command accepts it when --full-train is supplied.

Place the files as follows:

Practical-6/
└── data/
    ├── prepare_dataset.py
    └── raw/
        ├── KDDTrain+_20Percent.txt
        └── KDDTest+.txt

Prepare the classroom tables

From Practical-6, run:

python data/prepare_dataset.py

To use the full training set instead, place KDDTrain+.txt under data/raw/ and run:

python data/prepare_dataset.py --full-train

The utility:

• applies the official 41 feature names;
• adds attack_label, attack_group, binary_label, and difficulty;
• validates numeric and categorical columns;
• rejects unknown attack labels so mapping errors remain visible;
• preserves the original fixed train/test split;
• writes nsl_kdd_train.csv, nsl_kdd_test.csv, and feature_manifest.csv.

Quick verification

from pathlib import Path
import pandas as pd

train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))

print("Train:", train.shape)
print("Test:", test.shape)
print(train["binary_label"].value_counts().sort_index())
print(test["attack_group"].value_counts())

Each prepared table must have 41 model features plus these four audit/target columns:

attack_label, attack_group, binary_label, difficulty

Citation and redistribution

The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:

M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.

Retain this citation when redistributing prepared classroom data.

Important limitations

• NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
• The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
• Class prevalence does not represent a production SOC.
• IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
• The dataset has no dedicated botnet label.

---

Dataset setup — CTU-13 for Lab 7

CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional .binetflow file from one scenario; PCAP files and executable samples are not required.

Recommended download

Use the official CTU-13 dataset page and download Scenario 8, listed as CTU-Malware-Capture-Botnet-49. Locate this file inside the scenario's detailed-bidirectional-flow-labels directory:

capture20110816-3.binetflow

Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.

The official page also provides the complete CTU-13-Dataset.tar.bz2 archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.

Placement and preparation

Place the .binetflow file below data/raw/:

Practical-6/
└── data/
    ├── prepare_ctu13.py
    └── raw/
        └── capture20110816-3.binetflow

From Practical-6, run:

python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow

If exactly one .binetflow file exists below data/raw/, the input argument may be omitted:

python data/prepare_ctu13.py

The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:

python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000

It creates:

data/ctu13_botnet_flows.csv

The loader accepts both common CTU-13 flow schemas:

• the compact 15-column export containing Dir;
• the extended Argus export containing directional counters such as SrcPkts, DstPkts, and DstBytes but no Dir.

For the extended schema, the classroom direction feature is derived as bidirectional, source_only, destination_only, or unknown from the source and destination packet counters. When available, the original DstBytes value is used instead of reconstructing it from total and source bytes.

Label policy

• Botnet receives binary_label = 1.
• Normal receives binary_label = 0.
• Background receives binary_label = -1 because its status is uncertain.

Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.

Safety and leakage controls

• Use only .binetflow flow records; do not download or execute malware samples.
• Source and destination IP addresses are retained for analyst triage but excluded from model features.
• Raw label fields are excluded from model features.
• The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
• A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.

Citation

S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” Computers & Security, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011

CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.