CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,174 @@
|
||||
# Dataset setup — NSL-KDD
|
||||
|
||||
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
|
||||
|
||||
## Download
|
||||
|
||||
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
|
||||
|
||||
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
|
||||
|
||||
```text
|
||||
KDDTrain+_20Percent.txt
|
||||
KDDTest+.txt
|
||||
```
|
||||
|
||||
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
|
||||
|
||||
Place the files as follows:
|
||||
|
||||
```text
|
||||
Practical-6/
|
||||
└── data/
|
||||
├── prepare_dataset.py
|
||||
└── raw/
|
||||
├── KDDTrain+_20Percent.txt
|
||||
└── KDDTest+.txt
|
||||
```
|
||||
|
||||
## Prepare the classroom tables
|
||||
|
||||
From `Practical-6`, run:
|
||||
|
||||
```bash
|
||||
python data/prepare_dataset.py
|
||||
```
|
||||
|
||||
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
|
||||
|
||||
```bash
|
||||
python data/prepare_dataset.py --full-train
|
||||
```
|
||||
|
||||
The utility:
|
||||
|
||||
- applies the official 41 feature names;
|
||||
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
|
||||
- validates numeric and categorical columns;
|
||||
- rejects unknown attack labels so mapping errors remain visible;
|
||||
- preserves the original fixed train/test split;
|
||||
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
|
||||
|
||||
## Quick verification
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
import pandas as pd
|
||||
|
||||
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
|
||||
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
|
||||
|
||||
print("Train:", train.shape)
|
||||
print("Test:", test.shape)
|
||||
print(train["binary_label"].value_counts().sort_index())
|
||||
print(test["attack_group"].value_counts())
|
||||
```
|
||||
|
||||
Each prepared table must have 41 model features plus these four audit/target columns:
|
||||
|
||||
```text
|
||||
attack_label, attack_group, binary_label, difficulty
|
||||
```
|
||||
|
||||
## Citation and redistribution
|
||||
|
||||
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
|
||||
|
||||
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
|
||||
|
||||
Retain this citation when redistributing prepared classroom data.
|
||||
|
||||
## Important limitations
|
||||
|
||||
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
|
||||
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
|
||||
- Class prevalence does not represent a production SOC.
|
||||
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
|
||||
- The dataset has no dedicated botnet label.
|
||||
|
||||
---
|
||||
|
||||
# Dataset setup — CTU-13 for Lab 7
|
||||
|
||||
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
|
||||
|
||||
## Recommended download
|
||||
|
||||
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
|
||||
|
||||
```text
|
||||
capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
|
||||
|
||||
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
|
||||
|
||||
## Placement and preparation
|
||||
|
||||
Place the `.binetflow` file below `data/raw/`:
|
||||
|
||||
```text
|
||||
Practical-6/
|
||||
└── data/
|
||||
├── prepare_ctu13.py
|
||||
└── raw/
|
||||
└── capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
From `Practical-6`, run:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
|
||||
```
|
||||
|
||||
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py
|
||||
```
|
||||
|
||||
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
|
||||
|
||||
```bash
|
||||
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
|
||||
```
|
||||
|
||||
It creates:
|
||||
|
||||
```text
|
||||
data/ctu13_botnet_flows.csv
|
||||
```
|
||||
|
||||
The loader accepts both common CTU-13 flow schemas:
|
||||
|
||||
- the compact 15-column export containing `Dir`;
|
||||
- the extended Argus export containing directional counters such as
|
||||
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
|
||||
|
||||
For the extended schema, the classroom `direction` feature is derived as
|
||||
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
|
||||
and destination packet counters. When available, the original `DstBytes` value
|
||||
is used instead of reconstructing it from total and source bytes.
|
||||
|
||||
## Label policy
|
||||
|
||||
- `Botnet` receives `binary_label = 1`.
|
||||
- `Normal` receives `binary_label = 0`.
|
||||
- `Background` receives `binary_label = -1` because its status is uncertain.
|
||||
|
||||
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
|
||||
|
||||
## Safety and leakage controls
|
||||
|
||||
- Use only `.binetflow` flow records; do not download or execute malware samples.
|
||||
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
|
||||
- Raw label fields are excluded from model features.
|
||||
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
|
||||
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
|
||||
|
||||
## Citation
|
||||
|
||||
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
|
||||
|
||||
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,42 @@
|
||||
feature,feature_type,model_feature
|
||||
duration,numeric,True
|
||||
protocol_type,categorical,True
|
||||
service,categorical,True
|
||||
flag,categorical,True
|
||||
src_bytes,numeric,True
|
||||
dst_bytes,numeric,True
|
||||
land,numeric,True
|
||||
wrong_fragment,numeric,True
|
||||
urgent,numeric,True
|
||||
hot,numeric,True
|
||||
num_failed_logins,numeric,True
|
||||
logged_in,numeric,True
|
||||
num_compromised,numeric,True
|
||||
root_shell,numeric,True
|
||||
su_attempted,numeric,True
|
||||
num_root,numeric,True
|
||||
num_file_creations,numeric,True
|
||||
num_shells,numeric,True
|
||||
num_access_files,numeric,True
|
||||
num_outbound_cmds,numeric,True
|
||||
is_host_login,numeric,True
|
||||
is_guest_login,numeric,True
|
||||
count,numeric,True
|
||||
srv_count,numeric,True
|
||||
serror_rate,numeric,True
|
||||
srv_serror_rate,numeric,True
|
||||
rerror_rate,numeric,True
|
||||
srv_rerror_rate,numeric,True
|
||||
same_srv_rate,numeric,True
|
||||
diff_srv_rate,numeric,True
|
||||
srv_diff_host_rate,numeric,True
|
||||
dst_host_count,numeric,True
|
||||
dst_host_srv_count,numeric,True
|
||||
dst_host_same_srv_rate,numeric,True
|
||||
dst_host_diff_srv_rate,numeric,True
|
||||
dst_host_same_src_port_rate,numeric,True
|
||||
dst_host_srv_diff_host_rate,numeric,True
|
||||
dst_host_serror_rate,numeric,True
|
||||
dst_host_srv_serror_rate,numeric,True
|
||||
dst_host_rerror_rate,numeric,True
|
||||
dst_host_srv_rerror_rate,numeric,True
|
||||
|
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,261 @@
|
||||
"""Create a bounded, classroom-ready flow table from one CTU-13 .binetflow file."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RAW_DIR = HERE / "raw"
|
||||
|
||||
REQUIRED_RAW_COLUMNS = [
|
||||
"StartTime", "Dur", "Proto", "SrcAddr", "Sport", "DstAddr", "Dport",
|
||||
"State", "sTos", "dTos", "TotPkts", "TotBytes", "SrcBytes", "Label",
|
||||
]
|
||||
|
||||
# CTU-13 is distributed in more than one Argus text schema. The compact
|
||||
# 15-column export contains ``Dir``. The extended export supplied with some
|
||||
# scenarios omits it but provides per-direction packet and byte counters.
|
||||
OPTIONAL_RAW_COLUMNS = ["Dir", "SrcPkts", "DstPkts", "DstBytes"]
|
||||
|
||||
RENAME_MAP = {
|
||||
"StartTime": "start_time",
|
||||
"Dur": "duration",
|
||||
"Proto": "protocol",
|
||||
"SrcAddr": "src_addr",
|
||||
"Sport": "src_port",
|
||||
"Dir": "direction",
|
||||
"DstAddr": "dst_addr",
|
||||
"Dport": "dst_port",
|
||||
"State": "state",
|
||||
"sTos": "src_tos",
|
||||
"dTos": "dst_tos",
|
||||
"TotPkts": "total_packets",
|
||||
"TotBytes": "total_bytes",
|
||||
"SrcBytes": "src_bytes",
|
||||
"DstBytes": "dst_bytes",
|
||||
"Label": "raw_label",
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"input",
|
||||
nargs="?",
|
||||
type=Path,
|
||||
help="Path to one labeled CTU-13 .binetflow file. If omitted, data/raw is searched.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--per-class",
|
||||
type=int,
|
||||
default=20_000,
|
||||
help="Maximum sampled rows for each of Botnet, Normal, and Background.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chunk-size",
|
||||
type=int,
|
||||
default=200_000,
|
||||
help="Rows read at a time; reduce this value on low-memory computers.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=HERE / "ctu13_botnet_flows.csv",
|
||||
help="Prepared CSV destination.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_input(requested: Path | None) -> Path:
|
||||
if requested is not None:
|
||||
path = requested.expanduser().resolve()
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(f"CTU-13 input file not found: {path}")
|
||||
return path
|
||||
|
||||
matches = sorted(RAW_DIR.rglob("*.binetflow")) if RAW_DIR.exists() else []
|
||||
if len(matches) != 1:
|
||||
raise FileNotFoundError(
|
||||
f"Expected exactly one .binetflow file below {RAW_DIR}, found {len(matches)}. "
|
||||
"Pass the input path explicitly when more than one file is present."
|
||||
)
|
||||
return matches[0]
|
||||
|
||||
|
||||
def label_family(labels: pd.Series) -> pd.Series:
|
||||
text = labels.fillna("").astype(str)
|
||||
return pd.Series(
|
||||
np.select(
|
||||
[
|
||||
text.str.contains("Botnet", case=False, regex=False),
|
||||
text.str.contains("Normal", case=False, regex=False),
|
||||
text.str.contains("Background", case=False, regex=False),
|
||||
],
|
||||
["Botnet", "Normal", "Background"],
|
||||
default="Other",
|
||||
),
|
||||
index=labels.index,
|
||||
dtype="string",
|
||||
)
|
||||
|
||||
|
||||
def parse_port(values: pd.Series) -> pd.Series:
|
||||
text = values.fillna("").astype(str).str.strip().str.lower()
|
||||
decimal = pd.to_numeric(text.where(~text.str.startswith("0x")), errors="coerce")
|
||||
hexadecimal = text.where(text.str.startswith("0x")).map(
|
||||
lambda value: int(value, 16) if isinstance(value, str) and value else np.nan
|
||||
)
|
||||
return decimal.fillna(hexadecimal).astype("Float64")
|
||||
|
||||
|
||||
def inspect_schema(path: Path) -> tuple[list[str], str]:
|
||||
"""Validate a compact or extended CTU-13 header and select needed fields."""
|
||||
header = pd.read_csv(path, nrows=0).columns.tolist()
|
||||
missing = sorted(set(REQUIRED_RAW_COLUMNS) - set(header))
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"Missing required CTU-13 columns: {missing}. Available columns: {header}"
|
||||
)
|
||||
|
||||
selected = REQUIRED_RAW_COLUMNS + [
|
||||
column for column in OPTIONAL_RAW_COLUMNS if column in header
|
||||
]
|
||||
if "Dir" in header:
|
||||
direction_mode = "source Dir column"
|
||||
elif {"SrcPkts", "DstPkts"}.issubset(header):
|
||||
direction_mode = "derived from SrcPkts/DstPkts"
|
||||
else:
|
||||
direction_mode = "constant unknown (no Dir or packet-direction counters)"
|
||||
|
||||
print(f"Detected {len(header)}-column CTU-13 schema; direction is {direction_mode}.")
|
||||
return selected, direction_mode
|
||||
|
||||
|
||||
def ensure_direction(frame: pd.DataFrame) -> pd.DataFrame:
|
||||
"""Return a frame with a categorical Dir field for either CTU-13 schema."""
|
||||
if "Dir" in frame.columns:
|
||||
frame["Dir"] = frame["Dir"].fillna("unknown").astype(str).str.strip()
|
||||
frame.loc[frame["Dir"].eq(""), "Dir"] = "unknown"
|
||||
return frame
|
||||
|
||||
if {"SrcPkts", "DstPkts"}.issubset(frame.columns):
|
||||
src_packets = pd.to_numeric(frame["SrcPkts"], errors="coerce").fillna(0)
|
||||
dst_packets = pd.to_numeric(frame["DstPkts"], errors="coerce").fillna(0)
|
||||
frame["Dir"] = np.select(
|
||||
[
|
||||
(src_packets > 0) & (dst_packets > 0),
|
||||
(src_packets > 0) & (dst_packets <= 0),
|
||||
(src_packets <= 0) & (dst_packets > 0),
|
||||
],
|
||||
["bidirectional", "source_only", "destination_only"],
|
||||
default="unknown",
|
||||
)
|
||||
else:
|
||||
frame["Dir"] = "unknown"
|
||||
return frame
|
||||
|
||||
|
||||
def bounded_sample(path: Path, per_class: int, chunk_size: int) -> pd.DataFrame:
|
||||
if per_class < 1 or chunk_size < 1:
|
||||
raise ValueError("--per-class and --chunk-size must be positive integers.")
|
||||
|
||||
kept = {family: pd.DataFrame() for family in ("Botnet", "Normal", "Background")}
|
||||
rng = np.random.default_rng(42)
|
||||
saw_header = False
|
||||
selected_columns, _ = inspect_schema(path)
|
||||
|
||||
for chunk_number, chunk in enumerate(
|
||||
pd.read_csv(
|
||||
path,
|
||||
usecols=selected_columns,
|
||||
chunksize=chunk_size,
|
||||
low_memory=False,
|
||||
),
|
||||
start=1,
|
||||
):
|
||||
saw_header = True
|
||||
chunk = ensure_direction(chunk)
|
||||
output_columns = REQUIRED_RAW_COLUMNS + ["Dir"]
|
||||
if "DstBytes" in chunk.columns:
|
||||
output_columns.append("DstBytes")
|
||||
chunk = chunk[output_columns].copy()
|
||||
chunk["label_family"] = label_family(chunk["Label"])
|
||||
|
||||
for family in kept:
|
||||
part = chunk.loc[chunk["label_family"].eq(family)].copy()
|
||||
if part.empty:
|
||||
continue
|
||||
part["_sample_key"] = rng.random(len(part))
|
||||
candidates = pd.concat([kept[family], part], ignore_index=True)
|
||||
kept[family] = candidates.nsmallest(per_class, "_sample_key")
|
||||
|
||||
counts = {family: len(frame) for family, frame in kept.items()}
|
||||
print(f"Processed chunk {chunk_number}: retained {counts}")
|
||||
|
||||
if not saw_header:
|
||||
raise ValueError(f"No rows found in {path}")
|
||||
|
||||
sampled = pd.concat(kept.values(), ignore_index=True).drop(columns="_sample_key")
|
||||
if sampled.empty:
|
||||
raise ValueError("No Botnet, Normal, or Background labels were found.")
|
||||
return sampled
|
||||
|
||||
|
||||
def engineer_features(sampled: pd.DataFrame) -> pd.DataFrame:
|
||||
frame = sampled.rename(columns=RENAME_MAP).copy()
|
||||
frame["start_time"] = pd.to_datetime(frame["start_time"], errors="coerce")
|
||||
|
||||
numeric = [
|
||||
"duration", "src_tos", "dst_tos", "total_packets", "total_bytes", "src_bytes"
|
||||
]
|
||||
if "dst_bytes" in frame.columns:
|
||||
numeric.append("dst_bytes")
|
||||
for column in numeric:
|
||||
frame[column] = pd.to_numeric(frame[column], errors="coerce")
|
||||
|
||||
frame["src_port"] = parse_port(frame["src_port"])
|
||||
frame["dst_port"] = parse_port(frame["dst_port"])
|
||||
derived_dst_bytes = (frame["total_bytes"] - frame["src_bytes"]).clip(lower=0)
|
||||
if "dst_bytes" in frame.columns:
|
||||
frame["dst_bytes"] = frame["dst_bytes"].fillna(derived_dst_bytes).clip(lower=0)
|
||||
else:
|
||||
frame["dst_bytes"] = derived_dst_bytes
|
||||
frame["bytes_per_packet"] = frame["total_bytes"] / frame["total_packets"].clip(lower=1)
|
||||
frame["src_byte_ratio"] = frame["src_bytes"] / frame["total_bytes"].clip(lower=1)
|
||||
frame["start_hour"] = frame["start_time"].dt.hour
|
||||
frame["is_well_known_dst_port"] = frame["dst_port"].between(0, 1023).astype("Int64")
|
||||
frame["binary_label"] = frame["label_family"].map(
|
||||
{"Normal": 0, "Botnet": 1, "Background": -1}
|
||||
).astype(int)
|
||||
|
||||
ordered = [
|
||||
"start_time", "duration", "protocol", "src_addr", "src_port", "direction",
|
||||
"dst_addr", "dst_port", "state", "src_tos", "dst_tos", "total_packets",
|
||||
"total_bytes", "src_bytes", "dst_bytes", "bytes_per_packet", "src_byte_ratio",
|
||||
"start_hour", "is_well_known_dst_port", "raw_label", "label_family", "binary_label",
|
||||
]
|
||||
return frame[ordered].sort_values("start_time", na_position="last").reset_index(drop=True)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
input_path = find_input(args.input)
|
||||
output_path = args.output.expanduser().resolve()
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
sampled = bounded_sample(input_path, args.per_class, args.chunk_size)
|
||||
prepared = engineer_features(sampled)
|
||||
prepared.to_csv(output_path, index=False)
|
||||
|
||||
print(f"Created {output_path} with shape {prepared.shape}")
|
||||
print(prepared["label_family"].value_counts())
|
||||
print("Background is retained for triage but excluded from supervised ground-truth metrics.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Prepare fixed NSL-KDD train/test tables for Module 6 notebooks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
RAW_DIR = HERE / "raw"
|
||||
|
||||
FEATURE_NAMES = [
|
||||
"duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes",
|
||||
"land", "wrong_fragment", "urgent", "hot", "num_failed_logins",
|
||||
"logged_in", "num_compromised", "root_shell", "su_attempted", "num_root",
|
||||
"num_file_creations", "num_shells", "num_access_files", "num_outbound_cmds",
|
||||
"is_host_login", "is_guest_login", "count", "srv_count", "serror_rate",
|
||||
"srv_serror_rate", "rerror_rate", "srv_rerror_rate", "same_srv_rate",
|
||||
"diff_srv_rate", "srv_diff_host_rate", "dst_host_count",
|
||||
"dst_host_srv_count", "dst_host_same_srv_rate", "dst_host_diff_srv_rate",
|
||||
"dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate",
|
||||
"dst_host_serror_rate", "dst_host_srv_serror_rate", "dst_host_rerror_rate",
|
||||
"dst_host_srv_rerror_rate",
|
||||
]
|
||||
|
||||
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]
|
||||
|
||||
ATTACK_GROUPS = {
|
||||
"normal": "Normal",
|
||||
"back": "DoS", "land": "DoS", "neptune": "DoS", "pod": "DoS",
|
||||
"smurf": "DoS", "teardrop": "DoS", "apache2": "DoS", "mailbomb": "DoS",
|
||||
"processtable": "DoS", "udpstorm": "DoS", "worm": "DoS",
|
||||
"ipsweep": "Probe", "nmap": "Probe", "portsweep": "Probe", "satan": "Probe",
|
||||
"mscan": "Probe", "saint": "Probe",
|
||||
"ftp_write": "R2L", "guess_passwd": "R2L", "imap": "R2L",
|
||||
"multihop": "R2L", "phf": "R2L", "spy": "R2L", "warezclient": "R2L",
|
||||
"warezmaster": "R2L", "sendmail": "R2L", "named": "R2L",
|
||||
"snmpgetattack": "R2L", "snmpguess": "R2L", "xlock": "R2L",
|
||||
"xsnoop": "R2L", "httptunnel": "R2L",
|
||||
"buffer_overflow": "U2R", "loadmodule": "U2R", "perl": "U2R",
|
||||
"rootkit": "U2R", "ps": "U2R", "sqlattack": "U2R", "xterm": "U2R",
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--full-train",
|
||||
action="store_true",
|
||||
help="Use KDDTrain+.txt instead of KDDTrain+_20Percent.txt.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_one(filename: str) -> Path:
|
||||
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
|
||||
if len(matches) != 1:
|
||||
raise FileNotFoundError(
|
||||
f"Expected exactly one {filename!r} below {RAW_DIR}. "
|
||||
"Follow data/README.md and verify the extracted files."
|
||||
)
|
||||
return matches[0]
|
||||
|
||||
|
||||
def load_nsl_kdd(path: Path) -> pd.DataFrame:
|
||||
columns = FEATURE_NAMES + ["attack_label", "difficulty"]
|
||||
frame = pd.read_csv(path, header=None, names=columns, low_memory=False)
|
||||
frame["attack_label"] = (
|
||||
frame["attack_label"].astype(str).str.strip().str.rstrip(".").str.lower()
|
||||
)
|
||||
|
||||
unknown = sorted(set(frame["attack_label"]) - set(ATTACK_GROUPS))
|
||||
if unknown:
|
||||
raise ValueError(f"Unknown attack labels in {path.name}: {unknown}")
|
||||
|
||||
for column in CATEGORICAL_FEATURES:
|
||||
frame[column] = frame[column].astype(str).str.strip().str.lower()
|
||||
|
||||
numeric_features = [name for name in FEATURE_NAMES if name not in CATEGORICAL_FEATURES]
|
||||
for column in numeric_features + ["difficulty"]:
|
||||
frame[column] = pd.to_numeric(frame[column], errors="raise")
|
||||
|
||||
frame["attack_group"] = frame["attack_label"].map(ATTACK_GROUPS)
|
||||
frame["binary_label"] = (frame["attack_group"] != "Normal").astype(int)
|
||||
return frame[FEATURE_NAMES + ["attack_label", "attack_group", "binary_label", "difficulty"]]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
train_name = "KDDTrain+.txt" if args.full_train else "KDDTrain+_20Percent.txt"
|
||||
train = load_nsl_kdd(find_one(train_name))
|
||||
test = load_nsl_kdd(find_one("KDDTest+.txt"))
|
||||
|
||||
train_path = HERE / "nsl_kdd_train.csv"
|
||||
test_path = HERE / "nsl_kdd_test.csv"
|
||||
manifest_path = HERE / "feature_manifest.csv"
|
||||
|
||||
train.to_csv(train_path, index=False)
|
||||
test.to_csv(test_path, index=False)
|
||||
|
||||
manifest = pd.DataFrame({
|
||||
"feature": FEATURE_NAMES,
|
||||
"feature_type": [
|
||||
"categorical" if name in CATEGORICAL_FEATURES else "numeric"
|
||||
for name in FEATURE_NAMES
|
||||
],
|
||||
"model_feature": True,
|
||||
})
|
||||
manifest.to_csv(manifest_path, index=False)
|
||||
|
||||
print(f"Created {train_path} with shape {train.shape}")
|
||||
print(f"Created {test_path} with shape {test.shape}")
|
||||
print(f"Created {manifest_path}")
|
||||
print("Train attack groups:")
|
||||
print(train["attack_group"].value_counts())
|
||||
print("Test attack groups:")
|
||||
print(test["attack_group"].value_counts())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user