# Dataset setup — NSL-KDD

All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.

## Download

The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.

Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):

```text
KDDTrain+_20Percent.txt
KDDTest+.txt
```

A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.

Place the files as follows:

```text
Practical-6/
└── data/
    ├── prepare_dataset.py
    └── raw/
        ├── KDDTrain+_20Percent.txt
        └── KDDTest+.txt
```

## Prepare the classroom tables

From `Practical-6`, run:

```bash
python data/prepare_dataset.py
```

To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:

```bash
python data/prepare_dataset.py --full-train
```

The utility:

- applies the official 41 feature names;
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
- validates numeric and categorical columns;
- rejects unknown attack labels so mapping errors remain visible;
- preserves the original fixed train/test split;
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.

## Quick verification

```python
from pathlib import Path
import pandas as pd

train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))

print("Train:", train.shape)
print("Test:", test.shape)
print(train["binary_label"].value_counts().sort_index())
print(test["attack_group"].value_counts())
```

Each prepared table must have 41 model features plus these four audit/target columns:

```text
attack_label, attack_group, binary_label, difficulty
```

## Citation and redistribution

The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:

M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.

Retain this citation when redistributing prepared classroom data.

## Important limitations

- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
- Class prevalence does not represent a production SOC.
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
- The dataset has no dedicated botnet label.

---

# Dataset setup — CTU-13 for Lab 7

CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.

## Recommended download

Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:

```text
capture20110816-3.binetflow
```

Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.

The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.

## Placement and preparation

Place the `.binetflow` file below `data/raw/`:

```text
Practical-6/
└── data/
    ├── prepare_ctu13.py
    └── raw/
        └── capture20110816-3.binetflow
```

From `Practical-6`, run:

```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
```

If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:

```bash
python data/prepare_ctu13.py
```

The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:

```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
```

It creates:

```text
data/ctu13_botnet_flows.csv
```

The loader accepts both common CTU-13 flow schemas:

- the compact 15-column export containing `Dir`;
- the extended Argus export containing directional counters such as
  `SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.

For the extended schema, the classroom `direction` feature is derived as
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
and destination packet counters. When available, the original `DstBytes` value
is used instead of reconstructing it from total and source bytes.

## Label policy

- `Botnet` receives `binary_label = 1`.
- `Normal` receives `binary_label = 0`.
- `Background` receives `binary_label = -1` because its status is uncertain.

Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.

## Safety and leakage controls

- Use only `.binetflow` flow records; do not download or execute malware samples.
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
- Raw label fields are excluded from model features.
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.

## Citation

S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011

CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.