CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,174 @@
|
||||
# Dataset setup — NSL-KDD
|
||||
|
||||
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
|
||||
|
||||
## Download
|
||||
|
||||
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
|
||||
|
||||
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
|
||||
|
||||
```text
|
||||
KDDTrain+_20Percent.txt
|
||||
KDDTest+.txt
|
||||
```
|
||||
|
||||
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
|
||||
|
||||
Place the files as follows:
|
||||
|
||||
```text
|
||||
Practical-6/
|
||||
└── data/
|
||||
├── prepare_dataset.py
|
||||
└── raw/
|
||||
├── KDDTrain+_20Percent.txt
|
||||
└── KDDTest+.txt
|
||||
```
|
||||
|
||||
## Prepare the classroom tables
|
||||
|
||||
From `Practical-6`, run:
|
||||
|
||||
```bash
|
||||
python data/prepare_dataset.py
|
||||
```
|
||||
|
||||
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
|
||||
|
||||
```bash
|
||||
python data/prepare_dataset.py --full-train
|
||||
```
|
||||
|
||||
The utility:
|
||||
|
||||
- applies the official 41 feature names;
|
||||
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
|
||||
- validates numeric and categorical columns;
|
||||
- rejects unknown attack labels so mapping errors remain visible;
|
||||
- preserves the original fixed train/test split;
|
||||
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
|
||||
|
||||
## Quick verification
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
import pandas as pd
|
||||
|
||||
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
|
||||
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
|
||||
|
||||
print("Train:", train.shape)
|
||||
print("Test:", test.shape)
|
||||
print(train["binary_label"].value_counts().sort_index())
|
||||
print(test["attack_group"].value_counts())
|
||||
```
|
||||
|
||||
Each prepared table must have 41 model features plus these four audit/target columns:
|
||||
|
||||
```text
|
||||
attack_label, attack_group, binary_label, difficulty
|
||||
```
|
||||
|
||||
## Citation and redistribution
|
||||
|
||||
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
|
||||
|
||||
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
|
||||
|
||||
Retain this citation when redistributing prepared classroom data.
|
||||
|
||||
## Important limitations
|
||||
|
||||
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
|
||||
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
|
||||
- Class prevalence does not represent a production SOC.
|
||||
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
|
||||
- The dataset has no dedicated botnet label.
|
||||
|
||||
---
|
||||
|
||||
# Dataset setup — CTU-13 for Lab 7
|
||||
|
||||
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
|
||||
|
||||
## Recommended download
|
||||
|
||||
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
|
||||
|
||||
```text
|
||||
capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
|
||||
|
||||
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
|
||||
|
||||
## Placement and preparation
|
||||
|
||||
Place the `.binetflow` file below `data/raw/`:
|
||||
|
||||
```text
|
||||
Practical-6/
|
||||
└── data/
|
||||
├── prepare_ctu13.py
|
||||
└── raw/
|
||||
└── capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
From `Practical-6`, run:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py
|
||||
```
|
||||
|
||||
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
|
||||
```
|
||||
|
||||
It creates:
|
||||
|
||||
```text
|
||||
data/ctu13_botnet_flows.csv
|
||||
```
|
||||
|
||||
The loader accepts both common CTU-13 flow schemas:
|
||||
|
||||
- the compact 15-column export containing `Dir`;
|
||||
- the extended Argus export containing directional counters such as
|
||||
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
|
||||
|
||||
For the extended schema, the classroom `direction` feature is derived as
|
||||
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
|
||||
and destination packet counters. When available, the original `DstBytes` value
|
||||
is used instead of reconstructing it from total and source bytes.
|
||||
|
||||
## Label policy
|
||||
|
||||
- `Botnet` receives `binary_label = 1`.
|
||||
- `Normal` receives `binary_label = 0`.
|
||||
- `Background` receives `binary_label = -1` because its status is uncertain.
|
||||
|
||||
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
|
||||
|
||||
## Safety and leakage controls
|
||||
|
||||
- Use only `.binetflow` flow records; do not download or execute malware samples.
|
||||
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
|
||||
- Raw label fields are excluded from model features.
|
||||
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
|
||||
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
|
||||
|
||||
## Citation
|
||||
|
||||
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
|
||||
|
||||
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
|
||||
Reference in New Issue
Block a user