175 lines
6.3 KiB
Markdown
175 lines
6.3 KiB
Markdown
# Dataset setup — NSL-KDD
|
||
|
||
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
|
||
|
||
## Download
|
||
|
||
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
|
||
|
||
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
|
||
|
||
```text
|
||
KDDTrain+_20Percent.txt
|
||
KDDTest+.txt
|
||
```
|
||
|
||
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
|
||
|
||
Place the files as follows:
|
||
|
||
```text
|
||
Practical-6/
|
||
└── data/
|
||
├── prepare_dataset.py
|
||
└── raw/
|
||
├── KDDTrain+_20Percent.txt
|
||
└── KDDTest+.txt
|
||
```
|
||
|
||
## Prepare the classroom tables
|
||
|
||
From `Practical-6`, run:
|
||
|
||
```bash
|
||
python data/prepare_dataset.py
|
||
```
|
||
|
||
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
|
||
|
||
```bash
|
||
python data/prepare_dataset.py --full-train
|
||
```
|
||
|
||
The utility:
|
||
|
||
- applies the official 41 feature names;
|
||
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
|
||
- validates numeric and categorical columns;
|
||
- rejects unknown attack labels so mapping errors remain visible;
|
||
- preserves the original fixed train/test split;
|
||
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
|
||
|
||
## Quick verification
|
||
|
||
```python
|
||
from pathlib import Path
|
||
import pandas as pd
|
||
|
||
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
|
||
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
|
||
|
||
print("Train:", train.shape)
|
||
print("Test:", test.shape)
|
||
print(train["binary_label"].value_counts().sort_index())
|
||
print(test["attack_group"].value_counts())
|
||
```
|
||
|
||
Each prepared table must have 41 model features plus these four audit/target columns:
|
||
|
||
```text
|
||
attack_label, attack_group, binary_label, difficulty
|
||
```
|
||
|
||
## Citation and redistribution
|
||
|
||
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
|
||
|
||
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
|
||
|
||
Retain this citation when redistributing prepared classroom data.
|
||
|
||
## Important limitations
|
||
|
||
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
|
||
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
|
||
- Class prevalence does not represent a production SOC.
|
||
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
|
||
- The dataset has no dedicated botnet label.
|
||
|
||
---
|
||
|
||
# Dataset setup — CTU-13 for Lab 7
|
||
|
||
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
|
||
|
||
## Recommended download
|
||
|
||
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
|
||
|
||
```text
|
||
capture20110816-3.binetflow
|
||
```
|
||
|
||
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
|
||
|
||
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
|
||
|
||
## Placement and preparation
|
||
|
||
Place the `.binetflow` file below `data/raw/`:
|
||
|
||
```text
|
||
Practical-6/
|
||
└── data/
|
||
├── prepare_ctu13.py
|
||
└── raw/
|
||
└── capture20110816-3.binetflow
|
||
```
|
||
|
||
From `Practical-6`, run:
|
||
|
||
```bash
|
||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
|
||
```
|
||
|
||
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
|
||
|
||
```bash
|
||
python data/prepare_ctu13.py
|
||
```
|
||
|
||
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
|
||
|
||
```bash
|
||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
|
||
```
|
||
|
||
It creates:
|
||
|
||
```text
|
||
data/ctu13_botnet_flows.csv
|
||
```
|
||
|
||
The loader accepts both common CTU-13 flow schemas:
|
||
|
||
- the compact 15-column export containing `Dir`;
|
||
- the extended Argus export containing directional counters such as
|
||
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
|
||
|
||
For the extended schema, the classroom `direction` feature is derived as
|
||
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
|
||
and destination packet counters. When available, the original `DstBytes` value
|
||
is used instead of reconstructing it from total and source bytes.
|
||
|
||
## Label policy
|
||
|
||
- `Botnet` receives `binary_label = 1`.
|
||
- `Normal` receives `binary_label = 0`.
|
||
- `Background` receives `binary_label = -1` because its status is uncertain.
|
||
|
||
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
|
||
|
||
## Safety and leakage controls
|
||
|
||
- Use only `.binetflow` flow records; do not download or execute malware samples.
|
||
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
|
||
- Raw label fields are excluded from model features.
|
||
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
|
||
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
|
||
|
||
## Citation
|
||
|
||
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
|
||
|
||
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
|