CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets
This commit is contained in:
mikrotik
@@ -0,0 +1,227 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce4d0249",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 3 — Lab 1: Security Features and Binary Labels\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"**Topics:** samples, features, labels, class distribution\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c70b2987",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- identify samples, features, and labels in a security dataset;\n",
|
||||
"- inspect the schema and data types of a network-flow table;\n",
|
||||
"- convert multiple attack names into one binary target;\n",
|
||||
"- explain why class distribution matters before model training.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dd82f5a2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
|
||||
"RANDOM_STATE = 42\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6345844b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Load and inspect the flow table\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b8d885cb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH)\n",
|
||||
"\n",
|
||||
"print(\"Shape:\", df.shape)\n",
|
||||
"df.head()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "19535e33",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Identify the label and numeric features\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f87f05f8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"y_text = df[\"Label\"].astype(str).str.strip()\n",
|
||||
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
|
||||
"\n",
|
||||
"print(\"Feature matrix shape:\", X.shape)\n",
|
||||
"print(\"Original labels:\")\n",
|
||||
"print(y_text.value_counts())\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "59f3f674",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Create a binary security target\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b4d0c24e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"normalized_labels = y_text.str.strip().str.upper()\n",
|
||||
"y_binary = normalized_labels.ne(\"BENIGN\").astype(int)\n",
|
||||
"class_counts = y_binary.value_counts().sort_index()\n",
|
||||
"\n",
|
||||
"print(class_counts.rename(index={0: \"BENIGN\", 1: \"ATTACK\"}))\n",
|
||||
"print(\"Attack rate:\", round(y_binary.mean(), 3))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "28d40643",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Connect rows to feature vectors\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b0506346",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"sample_index = 0\n",
|
||||
"sample_features = X.iloc[sample_index]\n",
|
||||
"sample_target = int(y_binary.iloc[sample_index])\n",
|
||||
"\n",
|
||||
"print(\"Selected feature vector:\")\n",
|
||||
"print(sample_features)\n",
|
||||
"print(\"Binary target:\", sample_target)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3d4855d6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4159047f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert isinstance(df, pd.DataFrame)\n",
|
||||
"assert \"Label\" not in X.columns\n",
|
||||
"assert set(y_binary.unique()) == {0, 1}\n",
|
||||
"assert len(X) == len(y_binary)\n",
|
||||
"assert class_counts.loc[0] > class_counts.loc[1]\n",
|
||||
"assert len(sample_features) == X.shape[1]\n",
|
||||
"print(\"✓ Lab 1 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a66bcc4e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reflection questions\n",
|
||||
"\n",
|
||||
"1. Why is each row considered one sample while each column is considered one feature?\n",
|
||||
"2. What information is lost when several attack families are mapped to one `ATTACK` class?\n",
|
||||
"3. Why could high accuracy be misleading when `BENIGN` is the majority class?\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5b4575cc-2166-474b-bd0e-b49d94d1cfc2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5d03b205-5584-41c8-baaf-272b32bdf0b2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3ecfa6b2-cb34-45cc-98b4-5417f48f9f82",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "40ddd666-9f13-46bd-8a58-45bef952bfa2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8aacddc0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f26e6221",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- remove duplicate records and normalize invalid numeric values;\n",
|
||||
"- create a stratified train-test split;\n",
|
||||
"- fit imputation and scaling operations on training data only;\n",
|
||||
"- explain how fitting preprocessing on the test set causes leakage.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e440f789",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
|
||||
"RANDOM_STATE = 42\n",
|
||||
"\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.preprocessing import StandardScaler\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "28421713",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Load and clean the raw table\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "797874c4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"raw_df = pd.read_csv(DATA_PATH)\n",
|
||||
"raw_df.columns = raw_df.columns.str.strip()\n",
|
||||
"clean_df = raw_df.drop_duplicates().copy()\n",
|
||||
"clean_df = clean_df.replace([np.inf, -np.inf], np.nan)\n",
|
||||
"\n",
|
||||
"print(\"Raw rows:\", len(raw_df))\n",
|
||||
"print(\"Clean rows:\", len(clean_df))\n",
|
||||
"print(\"Missing numeric values:\", int(clean_df.select_dtypes(include=\"number\").isna().sum().sum()))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "17f8834a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Create features, target, and a stratified split\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3b0b8cf4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = clean_df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
|
||||
"y = clean_df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X,\n",
|
||||
" y,\n",
|
||||
" test_size=0.2,\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" stratify=y,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Train shape:\", X_train.shape)\n",
|
||||
"print(\"Test shape:\", X_test.shape)\n",
|
||||
"print(\"Train attack rate:\", round(y_train.mean(), 3))\n",
|
||||
"print(\"Test attack rate:\", round(y_test.mean(), 3))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a742c62a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Fit preprocessing on training data only\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "43a2c168",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"imputer = SimpleImputer(strategy=\"median\")\n",
|
||||
"scaler = StandardScaler()\n",
|
||||
"\n",
|
||||
"X_train_imputed = imputer.fit_transform(X_train)\n",
|
||||
"X_test_imputed = imputer.transform(X_test)\n",
|
||||
"\n",
|
||||
"X_train_scaled = scaler.fit_transform(X_train_imputed)\n",
|
||||
"X_test_scaled = scaler.transform(X_test_imputed)\n",
|
||||
"\n",
|
||||
"print(\"Mean of scaled training columns:\", np.round(X_train_scaled.mean(axis=0), 3))\n",
|
||||
"print(\"Mean of scaled test columns:\", np.round(X_test_scaled.mean(axis=0), 3))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0c3fc6ca",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Why the test mean is not exactly zero\n",
|
||||
"\n",
|
||||
"`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "459304bb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "478a41eb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert len(clean_df) <= len(raw_df)\n",
|
||||
"assert np.isfinite(X_train_scaled).all()\n",
|
||||
"assert np.isfinite(X_test_scaled).all()\n",
|
||||
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
|
||||
"assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
|
||||
"assert X_train_scaled.shape[1] == X.shape[1]\n",
|
||||
"print(\"✓ Lab 2 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ca45be08",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reflection questions\n",
|
||||
"\n",
|
||||
"1. Why must the imputer learn medians from the training set only?\n",
|
||||
"2. What information would leak if the scaler were fit before the train-test split?\n",
|
||||
"3. Why is stratification useful for an imbalanced security target?\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1f325e4e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 3 — Lab 3: A Random Forest Security Pipeline\n",
|
||||
"\n",
|
||||
"**Topics:** Scikit-learn Pipeline, imputation, Random Forest, fit, predict\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e9ba9f17",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- create one reusable preprocessing-and-model pipeline;\n",
|
||||
"- train a Random Forest binary classifier;\n",
|
||||
"- generate predictions for unseen data;\n",
|
||||
"- explain why tree models do not require feature standardization.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5dc04708",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
|
||||
"RANDOM_STATE = 42\n",
|
||||
"\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.ensemble import RandomForestClassifier\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3746fff2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Prepare the binary classification data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "96d91cc2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH)\n",
|
||||
"df.columns = df.columns.str.strip()\n",
|
||||
"df = df.drop_duplicates().replace([np.inf, -np.inf], np.nan)\n",
|
||||
"\n",
|
||||
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
|
||||
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X,\n",
|
||||
" y,\n",
|
||||
" test_size=0.2,\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" stratify=y,\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "69eb8c19",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Build one pipeline\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0dbae88b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pipeline = Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
|
||||
" (\"model\", RandomForestClassifier(\n",
|
||||
" n_estimators=100,\n",
|
||||
" max_depth=12,\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" n_jobs=-1,\n",
|
||||
" )),\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"print(pipeline)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ea59d402",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Why there is no `StandardScaler`\n",
|
||||
"\n",
|
||||
"Random Forest makes decisions by comparing feature values with thresholds. Scaling a column does not change the ordering of its values, so standardization is not required for this model family. The imputer remains inside the pipeline because the model needs finite values.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c8f9a97a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Train and predict\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c6194b17",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pipeline.fit(X_train, y_train)\n",
|
||||
"y_pred = pipeline.predict(X_test)\n",
|
||||
"\n",
|
||||
"print(\"Predictions:\", y_pred[:20])\n",
|
||||
"print(\"Number of test samples:\", len(y_pred))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b5ad6c87",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Predict one flow\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7eca199f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"one_flow = X_test.iloc[[0]]\n",
|
||||
"one_prediction = int(pipeline.predict(one_flow)[0])\n",
|
||||
"\n",
|
||||
"print(\"Prediction:\", one_prediction)\n",
|
||||
"print(\"Meaning:\", \"ATTACK\" if one_prediction == 1 else \"BENIGN\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90d72108",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "98f48da9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert list(pipeline.named_steps) == [\"imputer\", \"model\"]\n",
|
||||
"assert pipeline.named_steps[\"model\"].n_estimators == 100\n",
|
||||
"assert len(y_pred) == len(y_test)\n",
|
||||
"assert set(np.unique(y_pred)).issubset({0, 1})\n",
|
||||
"assert one_prediction in {0, 1}\n",
|
||||
"print(\"✓ Lab 3 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6100563d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reflection questions\n",
|
||||
"\n",
|
||||
"1. What problem does the imputer solve inside the pipeline?\n",
|
||||
"2. Why is a pipeline safer than manually transforming train and test data in separate cells?\n",
|
||||
"3. What does `random_state=42` contribute to the experiment?\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "aeab0fb6-54bf-4e88-87bc-400d658d67b7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "75b25828-0978-465b-a789-3181b0c25f38",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "8b198739-c8b0-4355-a44e-08f9c205768b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "10091acc-3fbd-4e91-9bb4-f994516ce995",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "76db28c7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 3 — Lab 4: Evaluate an Intrusion Classifier\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Topics:** confusion matrix, accuracy, precision, recall, F1-score, error analysis\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "778202bf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- calculate a confusion matrix and classification metrics;\n",
|
||||
"- distinguish false positives from false negatives;\n",
|
||||
"- explain why attack recall matters in intrusion detection;\n",
|
||||
"- inspect misclassified network flows.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "91e09f7c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
|
||||
"RANDOM_STATE = 42\n",
|
||||
"\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.ensemble import RandomForestClassifier\n",
|
||||
"from sklearn.metrics import (\n",
|
||||
" ConfusionMatrixDisplay, accuracy_score, classification_report,\n",
|
||||
" confusion_matrix, precision_score, recall_score, f1_score\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "29e0b207",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Train the baseline pipeline\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f53d7241",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
|
||||
"df.columns = df.columns.str.strip()\n",
|
||||
"df = df.replace([np.inf, -np.inf], np.nan)\n",
|
||||
"\n",
|
||||
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
|
||||
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"pipeline = Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
|
||||
" (\"model\", RandomForestClassifier(\n",
|
||||
" n_estimators=100, max_depth=12,\n",
|
||||
" random_state=RANDOM_STATE, n_jobs=-1\n",
|
||||
" )),\n",
|
||||
"])\n",
|
||||
"\n",
|
||||
"pipeline.fit(X_train, y_train)\n",
|
||||
"y_pred = pipeline.predict(X_test)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0781cb26",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Calculate the confusion matrix\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "21075b2f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cm = confusion_matrix(y_test, y_pred)\n",
|
||||
"tn, fp, fn, tp = cm.ravel()\n",
|
||||
"\n",
|
||||
"print(\"Confusion matrix:\\n\", cm)\n",
|
||||
"print(f\"TN={tn}, FP={fp}, FN={fn}, TP={tp}\")\n",
|
||||
"\n",
|
||||
"ConfusionMatrixDisplay(\n",
|
||||
" confusion_matrix=cm,\n",
|
||||
" display_labels=[\"BENIGN\", \"ATTACK\"],\n",
|
||||
").plot(cmap=\"Blues\")\n",
|
||||
"plt.title(\"Baseline intrusion classifier\")\n",
|
||||
"plt.show()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "223681c3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Calculate classification metrics\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a5a1597a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"accuracy = accuracy_score(y_test, y_pred)\n",
|
||||
"attack_precision = precision_score(y_test, y_pred, zero_division=0)\n",
|
||||
"attack_recall = recall_score(y_test, y_pred, zero_division=0)\n",
|
||||
"attack_f1 = f1_score(y_test, y_pred, zero_division=0)\n",
|
||||
"\n",
|
||||
"metrics = pd.Series({\n",
|
||||
" \"accuracy\": accuracy,\n",
|
||||
" \"attack_precision\": attack_precision,\n",
|
||||
" \"attack_recall\": attack_recall,\n",
|
||||
" \"attack_f1\": attack_f1,\n",
|
||||
"})\n",
|
||||
"print(metrics.round(3))\n",
|
||||
"\n",
|
||||
"print(classification_report(\n",
|
||||
" y_test,\n",
|
||||
" y_pred,\n",
|
||||
" target_names=[\"BENIGN\", \"ATTACK\"],\n",
|
||||
" zero_division=0,\n",
|
||||
"))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ebbfff71",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Inspect false negatives\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4adf798b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"results = X_test.copy()\n",
|
||||
"results[\"actual\"] = y_test\n",
|
||||
"results[\"predicted\"] = y_pred\n",
|
||||
"\n",
|
||||
"false_negatives = results[\n",
|
||||
" (results[\"actual\"] == 1) & (results[\"predicted\"] == 0)\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"print(\"False negatives:\", len(false_negatives))\n",
|
||||
"false_negatives.head()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4cba4276",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0a64652b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert cm.shape == (2, 2)\n",
|
||||
"assert tn + fp + fn + tp == len(y_test)\n",
|
||||
"assert 0 <= accuracy <= 1\n",
|
||||
"assert 0 <= attack_precision <= 1\n",
|
||||
"assert 0 <= attack_recall <= 1\n",
|
||||
"assert 0 <= attack_f1 <= 1\n",
|
||||
"assert len(false_negatives) == fn\n",
|
||||
"print(\"✓ Lab 4 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f5b93d9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Analyst questions\n",
|
||||
"\n",
|
||||
"1. Which confusion-matrix cell represents a missed attack?\n",
|
||||
"2. Which metric should receive more attention if missed attacks are very costly?\n",
|
||||
"3. Why can improving attack recall increase the number of false alerts?\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3a90183",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Module 3 — Lab 5: A Controlled Class-Imbalance Experiment\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"**Topics:** class imbalance, class weights, controlled comparison, metric trade-offs\n",
|
||||
"\n",
|
||||
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4d80ad41",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Learning objectives\n",
|
||||
"\n",
|
||||
"After completing this lab, you will be able to:\n",
|
||||
"\n",
|
||||
"- measure the class imbalance in a security dataset;\n",
|
||||
"- compare two runs of the same model on the same split;\n",
|
||||
"- use `class_weight=\"balanced\"` without changing the rest of the workflow;\n",
|
||||
"- select a model configuration using attack precision, recall, and F1.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "61a4b369",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pathlib import Path\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"\n",
|
||||
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
|
||||
"RANDOM_STATE = 42\n",
|
||||
"\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"from sklearn.impute import SimpleImputer\n",
|
||||
"from sklearn.ensemble import RandomForestClassifier\n",
|
||||
"from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1301b149",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Prepare one fixed experiment\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f5ea362d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
|
||||
"df.columns = df.columns.str.strip()\n",
|
||||
"df = df.replace([np.inf, -np.inf], np.nan)\n",
|
||||
"\n",
|
||||
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
|
||||
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"benign_count = int((y_train == 0).sum())\n",
|
||||
"attack_count = int((y_train == 1).sum())\n",
|
||||
"imbalance_ratio = benign_count / attack_count\n",
|
||||
"\n",
|
||||
"print(f\"BENIGN={benign_count}, ATTACK={attack_count}\")\n",
|
||||
"print(f\"BENIGN-to-ATTACK ratio: {imbalance_ratio:.2f}:1\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e3dfcb43",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Define a reusable training function\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "51c588dd",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_and_measure(class_weight):\n",
|
||||
" model = Pipeline([\n",
|
||||
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
|
||||
" (\"model\", RandomForestClassifier(\n",
|
||||
" n_estimators=100,\n",
|
||||
" max_depth=12,\n",
|
||||
" class_weight=class_weight,\n",
|
||||
" random_state=RANDOM_STATE,\n",
|
||||
" n_jobs=-1,\n",
|
||||
" )),\n",
|
||||
" ])\n",
|
||||
" model.fit(X_train, y_train)\n",
|
||||
" predictions = model.predict(X_test)\n",
|
||||
" tn, fp, fn, tp = confusion_matrix(y_test, predictions).ravel()\n",
|
||||
" return {\n",
|
||||
" \"precision\": precision_score(y_test, predictions, zero_division=0),\n",
|
||||
" \"recall\": recall_score(y_test, predictions, zero_division=0),\n",
|
||||
" \"f1\": f1_score(y_test, predictions, zero_division=0),\n",
|
||||
" \"false_positives\": int(fp),\n",
|
||||
" \"false_negatives\": int(fn),\n",
|
||||
" }\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c81e9645",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Run the controlled comparison\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ac3530d7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"baseline_metrics = train_and_measure(class_weight=None)\n",
|
||||
"balanced_metrics = train_and_measure(class_weight=\"balanced\")\n",
|
||||
"\n",
|
||||
"comparison_df = pd.DataFrame(\n",
|
||||
" [baseline_metrics, balanced_metrics],\n",
|
||||
" index=[\"Baseline\", \"Balanced\"],\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"comparison_df.round(3)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e4f99ffd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Make an evidence-based recommendation\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "fc7c44fb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"recommended_configuration = comparison_df[\"f1\"].idxmax()\n",
|
||||
"\n",
|
||||
"print(\"Recommended configuration:\", recommended_configuration)\n",
|
||||
"print(comparison_df.loc[recommended_configuration].round(3))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c6e2a54c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Self-check\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c6389320",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"assert benign_count > attack_count\n",
|
||||
"assert imbalance_ratio > 1\n",
|
||||
"assert list(comparison_df.index) == [\"Baseline\", \"Balanced\"]\n",
|
||||
"assert {\"precision\", \"recall\", \"f1\", \"false_positives\", \"false_negatives\"}.issubset(comparison_df.columns)\n",
|
||||
"assert comparison_df[[\"precision\", \"recall\", \"f1\"]].apply(lambda col: col.between(0, 1).all()).all()\n",
|
||||
"assert recommended_configuration in {\"Baseline\", \"Balanced\"}\n",
|
||||
"print(\"✓ Lab 5 completed\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "69c051ee",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Analyst questions\n",
|
||||
"\n",
|
||||
"1. Did class weighting improve attack recall in this run?\n",
|
||||
"2. What happened to false positives and false negatives?\n",
|
||||
"3. Would you choose the same configuration if analyst time were extremely limited?\n",
|
||||
"4. Why is changing only one variable important in a controlled experiment?\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user