mikrotik/labs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

Browse Source
This commit is contained in:
mikrotik
2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+3
View File
@@ -0,0 +1,3 @@
.ipynb_checkpoints/
__pycache__/
*.pyc
Lab-2/Lab-2-Practical.zip
BIN
View File
Binary file not shown.
Lab-2/Lab-2-Practical/01_Python_Basics_Lab.ipynb
+241
View File
@@ -0,0 +1,241 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "eca32fd8",
"metadata": {},
"source": [
"# Module 2 — Lab 1: Python Fundamentals for Security Analysis\n",
"\n",
"**Topics:** variables, data types, strings, type casting, operators\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "da8ed50e",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- store security information in variables and select suitable data types;\n",
"- normalize log strings using string methods;\n",
"- cast values read from logs to the required data types;\n",
"- combine comparison and logical operators to flag suspicious activity.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0a4d2b7d",
"metadata": {},
"outputs": [],
"source": [
"print(\"Hello, Cyber World!\")\n",
"\n",
"ip_address = \"198.51.100.14\"\n",
"port_number = 443\n",
"failed_logins = 7\n",
"response_time = 0.125\n",
"is_admin = False\n",
"\n",
"print(f\"IP: {ip_address} | Port: {port_number}\")\n",
"print(type(ip_address), type(port_number), type(response_time), type(is_admin))\n"
]
},
{
"cell_type": "markdown",
"id": "d700974b",
"metadata": {},
"source": [
"## 1. Strings in security logs\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "776c60e6",
"metadata": {},
"outputs": [],
"source": [
"log_message = \" FAILED login for Admin from 198.51.100.14 \"\n",
"url = \"https://example.com/login\"\n",
"\n",
"print(log_message.strip())\n",
"print(log_message.lower())\n",
"print(log_message.replace(\"FAILED\", \"DENIED\"))\n",
"print(url[0:5])\n"
]
},
{
"cell_type": "markdown",
"id": "a31e697c",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Normalize a username and extract the protocol\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "02195f3c",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"raw_username = \" AdMin \"\n",
"\n",
"# TODO 1: remove surrounding whitespace and convert the username to lowercase\n",
"normalized_username = None\n",
"\n",
"url = \"https://example.com/login\"\n",
"# TODO 2: use slicing to extract the string \"https\"\n",
"protocol = None\n",
"\n",
"print(normalized_username)\n",
"print(protocol)\n"
]
},
{
"cell_type": "markdown",
"id": "c22bc685",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Type casting and operators\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ee235adb",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"port_text = \"8080\"\n",
"# TODO 3: cast port_text to int\n",
"port_number = None\n",
"# TODO 4: calculate the next port number\n",
"next_port = None\n",
"\n",
"login_attempts = 7\n",
"user_ip = \"198.51.100.14\"\n",
"# TODO 5: flag as suspicious when attempts > 5 AND the IP is not localhost\n",
"is_suspicious = None\n",
"\n",
"print(f\"Next port: {next_port}\")\n",
"print(f\"Suspicious: {is_suspicious}\")\n"
]
},
{
"cell_type": "markdown",
"id": "8bfdaf0b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Calculate a basic risk score\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1dde9681",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"failed_logins = 7\n",
"is_admin = False\n",
"is_off_hours = True\n",
"\n",
"# Rule: each failure adds 10 points, admin adds 25, and off-hours adds 15\n",
"# TODO 6: calculate risk_score and require review at 60 points or higher\n",
"risk_score = None\n",
"requires_review = None\n",
"\n",
"print(f\"Risk score: {risk_score}\")\n",
"print(f\"Requires review: {requires_review}\")\n"
]
},
{
"cell_type": "markdown",
"id": "bdfa1e23",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e9fa7c6",
"metadata": {},
"outputs": [],
"source": [
"assert normalized_username == \"admin\"\n",
"assert protocol == \"https\"\n",
"assert next_port == 8081\n",
"assert is_suspicious is True\n",
"assert risk_score == 85\n",
"assert requires_review is True\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "c63bef39",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why does `port_text + 1` fail while `int(port_text) + 1` works?\n",
"2. What changes if you replace `and` with `or` in the `is_suspicious` condition?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/02_Control_Flow_Functions_Lab.ipynb
+224
View File
@@ -0,0 +1,224 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d9c31a26",
"metadata": {},
"source": [
"# Module 2 — Lab 2: Control Flow and Functions\n",
"\n",
"**Topics:** if/elif/else, for, while, break, continue, function\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "e060cb80",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"You need to build small rules that classify login alerts and automatically review a list of IP addresses. All activity is simulated with local data; this notebook does not connect to or scan any real system.\n"
]
},
{
"cell_type": "markdown",
"id": "6d919cf5",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Classify failed login attempts\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ef271740",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"failed_attempts = 7\n",
"\n",
"# TODO 1: assign severity using the thresholds 10, 5, 1, and 0\n",
"severity = None\n",
"\n",
"print(severity)\n"
]
},
{
"cell_type": "markdown",
"id": "92cd839f",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Review IP addresses with `for`, `continue`, and `break`\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1b87cad",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"blocked_ips = {\"10.0.0.5\", \"10.0.0.12\"}\n",
"targets = [f\"10.0.0.{i}\" for i in range(1, 16)]\n",
"reviewed = []\n",
"\n",
"for ip in targets:\n",
" last_octet = int(ip.split(\".\")[-1])\n",
"\n",
" # TODO 2: skip IP addresses whose final octet is odd\n",
" pass\n",
"\n",
" # TODO 3: append the IP address to reviewed\n",
" pass\n",
"\n",
" # TODO 4: if the IP is in blocked_ips, print an alert and stop the loop\n",
" pass\n",
"\n",
"print(\"Reviewed:\", reviewed)\n"
]
},
{
"cell_type": "markdown",
"id": "2128332a",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Limit retry attempts with `while`\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4fb4787e",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"attempt = 0\n",
"max_attempts = 3\n",
"service_responded = False\n",
"\n",
"# TODO 5: loop while max_attempts has not been reached and the service has not responded.\n",
"# Simulate the service responding on the second attempt.\n",
"while False:\n",
" pass\n",
"\n",
"print(f\"Service responded: {service_responded}\")\n"
]
},
{
"cell_type": "markdown",
"id": "9fc142dd",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Package a rule in a function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "63e40cf6",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"def classify_login_risk(failed_attempts, is_admin=False, is_off_hours=False):\n",
" # Return LOW, MEDIUM, HIGH, or CRITICAL for a login event.\n",
" # TODO 6: calculate the score and return a label using thresholds 100, 60, and 30\n",
" pass\n",
"\n",
"\n",
"print(classify_login_risk(2))\n",
"print(classify_login_risk(7, is_off_hours=True))\n",
"print(classify_login_risk(8, is_admin=True))\n"
]
},
{
"cell_type": "markdown",
"id": "a10e84bf",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b1d43daf",
"metadata": {},
"outputs": [],
"source": [
"assert severity == \"HIGH\"\n",
"assert reviewed == [\"10.0.0.2\", \"10.0.0.4\", \"10.0.0.6\", \"10.0.0.8\", \"10.0.0.10\", \"10.0.0.12\"]\n",
"assert attempt == 2 and service_responded is True\n",
"assert classify_login_risk(2) == \"LOW\"\n",
"assert classify_login_risk(7, is_off_hours=True) == \"HIGH\"\n",
"assert classify_login_risk(8, is_admin=True) == \"CRITICAL\"\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "3f5385ff",
"metadata": {},
"source": [
"## Extension challenge\n",
"\n",
"Modify the function to add 20 points when `source_ip` belongs to a set of blocked IP addresses. Write at least two `assert` statements for the new rule.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/03_Data_Structures_Log_Parsing_Lab.ipynb
+234
View File
@@ -0,0 +1,234 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "e58bd80a",
"metadata": {},
"source": [
"# Module 2 — Lab 3: Data Structures and Raw Log Analysis\n",
"\n",
"**Topics:** list, tuple, dictionary, set, parsing\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "53230dd3",
"metadata": {},
"source": [
"## Input data\n",
"\n",
"Each log line contains four comma-separated fields: `timestamp,source_ip,username,status`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad6600dc",
"metadata": {},
"outputs": [],
"source": [
"raw_logs = [\n",
" \"2026-09-01T08:00:00Z,203.0.113.55,alice,SUCCESS\",\n",
" \"2026-09-01T08:01:00Z,198.51.100.14,admin,FAILED\",\n",
" \"2026-09-01T08:01:20Z,198.51.100.14,root,FAILED\",\n",
" \"2026-09-01T08:02:00Z,192.0.2.88,bob,SUCCESS\",\n",
" \"2026-09-01T08:02:30Z,198.51.100.14,guest,FAILED\",\n",
" \"2026-09-01T08:03:00Z,192.0.2.99,carol,FAILED\",\n",
"]\n",
"\n",
"critical_fields = (\"timestamp\", \"source_ip\", \"username\", \"status\")\n",
"blocked_ips = {\"198.51.100.14\", \"203.0.113.200\"}\n",
"\n",
"print(f\"Rows: {len(raw_logs)}\")\n",
"print(\"Critical fields:\", critical_fields)\n"
]
},
{
"cell_type": "markdown",
"id": "b99fcf3b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Convert one log line into a dictionary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a7f7c548",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"def parse_log_line(line):\n",
" # TODO 1: split the line into four variables\n",
" # TODO 2: return a dictionary with the four keys in critical_fields\n",
" # Normalize username to lowercase and status to uppercase.\n",
" pass\n",
"\n",
"\n",
"sample_event = parse_log_line(raw_logs[0])\n",
"print(sample_event)\n"
]
},
{
"cell_type": "markdown",
"id": "0d95749d",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Create an event list and a set of unique IP addresses\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "809d56f4",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 3: create the events list by parsing each raw log line\n",
"events = []\n",
"\n",
"# TODO 4: create the unique_ips set from events\n",
"unique_ips = set()\n",
"\n",
"print(\"Unique IPs:\", sorted(unique_ips))\n",
"print(\"Number of unique IPs:\", len(unique_ips))\n"
]
},
{
"cell_type": "markdown",
"id": "f97d6524",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Count failed logins with a dictionary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7bb93fd8",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"failed_by_ip = {}\n",
"\n",
"# TODO 5: count FAILED events for each source_ip.\n",
"# Hint: dictionary.get(ip, 0)\n",
"\n",
"print(failed_by_ip)\n"
]
},
{
"cell_type": "markdown",
"id": "1b67dd02",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Combine dictionaries and sets to create alerts\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7a77c94f",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"alerts = []\n",
"\n",
"# TODO 6: create an alert when count >= 3 OR the IP is in blocked_ips.\n",
"# Each alert is a dictionary containing source_ip, failed_attempts, and blocked.\n",
"\n",
"print(alerts)\n"
]
},
{
"cell_type": "markdown",
"id": "9f495105",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "487170fc",
"metadata": {},
"outputs": [],
"source": [
"assert sample_event[\"username\"] == \"alice\"\n",
"assert len(events) == 6\n",
"assert unique_ips == {\"203.0.113.55\", \"198.51.100.14\", \"192.0.2.88\", \"192.0.2.99\"}\n",
"assert failed_by_ip[\"198.51.100.14\"] == 3\n",
"assert alerts == [{\"source_ip\": \"198.51.100.14\", \"failed_attempts\": 3, \"blocked\": True}]\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "f634dc9c",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"- Why is a `set` suitable for storing unique IP addresses or a blocklist?\n",
"- Why is a `tuple` suitable for field names that should not change?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/04_NumPy_Security_Metrics_Lab.ipynb
+204
View File
@@ -0,0 +1,204 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "2444933d",
"metadata": {},
"source": [
"# Module 2 — Lab 4: NumPy for Security Metrics\n",
"\n",
"**Topics:** array, vectorization, statistics, boolean mask\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "0f43f1b4",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"A network sensor reports packet sizes, latency, and failed authentication attempts for each time window. We use NumPy to calculate efficiently across the entire array.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0b917960",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"packet_sizes = np.array([64, 128, 1500, 512, 1514, 256, 1400, 80])\n",
"latency_ms = np.array([12.5, 18.0, 250.0, 33.5, 180.0, 25.0, 95.0, 10.0])\n",
"failed_attempts = np.array([0, 1, 8, 2, 10, 1, 6, 0])\n",
"\n",
"print(packet_sizes.shape, packet_sizes.dtype)\n"
]
},
{
"cell_type": "markdown",
"id": "ca8c8c19",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Unit conversion and descriptive statistics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "01b704ef",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 1: convert packet_sizes from bytes to KB with a vectorized operation\n",
"packet_kb = None\n",
"\n",
"# TODO 2: calculate the mean and 95th percentile of latency_ms\n",
"mean_latency = None\n",
"p95_latency = None\n",
"\n",
"print(\"Packet sizes (KB):\", packet_kb)\n",
"print(\"Mean latency:\", mean_latency)\n",
"print(\"P95 latency:\", p95_latency)\n"
]
},
{
"cell_type": "markdown",
"id": "34dfe8cd",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Filter with a Boolean mask\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "070dc425",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 3: True when size > 1400 OR latency > 150 OR failures >= 8\n",
"suspicious_mask = None\n",
"\n",
"# TODO 4: obtain the indices matching the mask with np.where\n",
"suspicious_indices = None\n",
"\n",
"print(\"Mask:\", suspicious_mask)\n",
"print(\"Suspicious indices:\", suspicious_indices)\n"
]
},
{
"cell_type": "markdown",
"id": "d5e6d529",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Normalize values and create a risk score\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f126015",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"def min_max(values):\n",
" values = np.asarray(values, dtype=float)\n",
" # TODO 5: return a zero array when max == min; otherwise apply min-max normalization\n",
" pass\n",
"\n",
"\n",
"latency_score = min_max(latency_ms)\n",
"failure_score = min_max(failed_attempts)\n",
"\n",
"# TODO 6: calculate 40% latency_score + 60% failure_score\n",
"risk_score = None\n",
"\n",
"print(\"Risk scores:\", risk_score)\n",
"print(\"Highest-risk index:\", None if risk_score is None else int(np.argmax(risk_score)))\n"
]
},
{
"cell_type": "markdown",
"id": "a4dcff69",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a329c7f0",
"metadata": {},
"outputs": [],
"source": [
"assert packet_sizes.shape == (8,)\n",
"assert np.isclose(mean_latency, 78.0)\n",
"assert suspicious_indices.tolist() == [2, 4]\n",
"assert int(np.argmax(risk_score)) == 4\n",
"assert np.all((risk_score >= 0) & (risk_score <= 1))\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "915b1cef",
"metadata": {},
"source": [
"## Extension challenge\n",
"\n",
"Change the weights assigned to latency and failed attempts. Observe whether the highest-risk element changes and explain why.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/05_Pandas_Log_Cleaning_Lab.ipynb
+259
View File
@@ -0,0 +1,259 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "e51dfbb6",
"metadata": {},
"source": [
"# Module 2 — Lab 5: Pandas — Loading, Inspecting, and Cleaning Logs\n",
"\n",
"\n",
"**Topics:** CSV, DataFrame, missing values, duplicates, datetime, filtering\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "e505f19b",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"The `data/auth_log.csv` file contains duplicate rows, invalid timestamps, missing critical fields, and inconsistent capitalization. The goal is to produce a clean DataFrame ready for analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4a8981a",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/auth_log.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"\n",
"log_df = pd.read_csv(DATA_PATH)\n",
"log_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "5aa72689",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Inspect data quality\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9bcc9310",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 1: print the shape, duplicate count, missing-value counts, and dtypes\n",
"print(\"Shape:\", None)\n",
"print(\"Duplicate rows:\", None)\n",
"print(\"Missing values by column:\")\n",
"print(None)\n",
"print(\"\\nData types:\")\n",
"print(None)\n"
]
},
{
"cell_type": "markdown",
"id": "3c13cc30",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Remove duplicates and normalize timestamps and critical fields\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7dc3dbd3",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"clean_df = log_df.copy()\n",
"\n",
"# TODO 2: remove duplicate rows\n",
"\n",
"# TODO 3: parse timestamp with pd.to_datetime using errors=\"coerce\" and utc=True\n",
"\n",
"# TODO 4: drop rows missing timestamp, source_ip, or status\n"
]
},
{
"cell_type": "markdown",
"id": "95d39a43",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Normalize text and create time-based fields\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9e434ec2",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 5: strip surrounding whitespace from source_ip\n",
"# TODO 6: fill missing usernames with \"unknown\", strip whitespace, and convert to lowercase\n",
"# TODO 7: strip whitespace and convert status to uppercase\n",
"\n",
"# TODO 8: retain only SUCCESS and FAILED status values\n",
"# TODO 9: create the hour column from timestamp\n",
"# TODO 10: sort by timestamp and reset the index\n",
"\n",
"clean_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "94783144",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Filter and aggregate\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ef6981bc",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 11: filter FAILED rows\n",
"failed_df = None\n",
"\n",
"# TODO 12: count FAILED events by source_ip\n",
"failed_by_ip = None\n",
"\n",
"# TODO 13: count status values\n",
"status_counts = None\n",
"\n",
"print(\"Status counts:\")\n",
"print(status_counts)\n",
"print(\"\\nFailed attempts by IP:\")\n",
"print(failed_by_ip)\n"
]
},
{
"cell_type": "markdown",
"id": "d8ccd176",
"metadata": {},
"source": [
"## 5. Export clean data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51b00290",
"metadata": {},
"outputs": [],
"source": [
"clean_df.to_csv(OUTPUT_DIR / \"clean_auth_log.csv\", index=False)\n",
"clean_df.to_json(\n",
" OUTPUT_DIR / \"clean_auth_log.json\",\n",
" orient=\"records\",\n",
" indent=2,\n",
" date_format=\"iso\",\n",
")\n",
"failed_by_ip.to_csv(\n",
" OUTPUT_DIR / \"failed_login_by_ip.csv\",\n",
" header=True,\n",
")\n",
"\n",
"print(\"Exported files to:\", OUTPUT_DIR.resolve())\n"
]
},
{
"cell_type": "markdown",
"id": "622a8262",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c5d54f7",
"metadata": {},
"outputs": [],
"source": [
"assert log_df.duplicated().sum() >= 1\n",
"assert clean_df.duplicated().sum() == 0\n",
"assert clean_df[\"timestamp\"].isna().sum() == 0\n",
"assert set(clean_df[\"status\"]) <= {\"SUCCESS\", \"FAILED\"}\n",
"assert clean_df[\"username\"].str.strip().eq(clean_df[\"username\"]).all()\n",
"assert failed_by_ip.index[0] == \"198.51.100.14\"\n",
"assert (OUTPUT_DIR / \"clean_auth_log.csv\").exists()\n",
"assert (OUTPUT_DIR / \"clean_auth_log.json\").exists()\n",
"print(\"✓ Lab 5 completed\")\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/06_Capstone_Auth_Log_Analysis_Lab.ipynb
+284
View File
@@ -0,0 +1,284 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "3d9865d1",
"metadata": {},
"source": [
"# Module 2 — Lab 6: Capstone — Investigating Suspicious Logins\n",
"\n",
" \n",
"**Topics:** Pandas, NumPy, Matplotlib, export, analyst summary\n",
"\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "dda323ff",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"As a junior security analyst, you must build a complete workflow:\n",
"\n",
"1. load and clean authentication logs;\n",
"2. identify IP addresses with repeated failed logins;\n",
"3. identify off-hours activity;\n",
"4. visualize the results;\n",
"5. export the data and write a short conclusion.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7b91b90f",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/auth_log.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n"
]
},
{
"cell_type": "markdown",
"id": "2e73090b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Write a reusable cleaning function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21e69c93",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"def clean_auth_log(df):\n",
" result = df.copy()\n",
" # TODO 1: drop duplicate rows\n",
" # TODO 2: parse timestamp using errors=\"coerce\" and utc=True\n",
" # TODO 3: drop rows missing timestamp, source_ip, or status\n",
" # TODO 4: normalize source_ip, username, and status\n",
" # TODO 5: retain SUCCESS and FAILED records only\n",
" # TODO 6: create hour and is_off_hours (< 6 or >= 22)\n",
" # TODO 7: sort by timestamp and reset the index\n",
" return result\n",
"\n",
"\n",
"raw_df = pd.read_csv(DATA_PATH)\n",
"clean_df = clean_auth_log(raw_df)\n",
"print(f\"Raw rows: {len(raw_df)} | Clean rows: {len(clean_df)}\")\n",
"clean_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "57d60f86",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Analyze failures and rank IP addresses\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1ea702f5",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 8: filter FAILED records\n",
"failed_df = None\n",
"\n",
"# TODO 9: group by source_ip, count, sort descending, and name the Series failed_attempts\n",
"failed_by_ip = None\n",
"\n",
"THRESHOLD = 3\n",
"# TODO 10: retain IP addresses with count >= THRESHOLD\n",
"suspicious_ips = None\n",
"\n",
"print(failed_by_ip)\n",
"print(suspicious_ips)\n"
]
},
{
"cell_type": "markdown",
"id": "eba0fd3d",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Calculate additional investigation metrics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fea68f16",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 11: create status_counts in SUCCESS, FAILED order\n",
"status_counts = None\n",
"\n",
"# TODO 12: filter off-hours failed events\n",
"off_hours_failed = None\n",
"\n",
"# TODO 13: determine top_ip, top_ip_count, and failure_rate (%)\n",
"top_ip = None\n",
"top_ip_count = None\n",
"failure_rate = None\n",
"\n",
"print(f\"Top suspicious IP: {top_ip} ({top_ip_count} failures)\")\n",
"print(f\"Failure rate: {failure_rate}%\")\n",
"print(\"Off-hours failed attempts:\", None if off_hours_failed is None else len(off_hours_failed))\n"
]
},
{
"cell_type": "markdown",
"id": "eb9420ec",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Visualize the findings\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f438336",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 14: create a figure with two charts:\n",
"# - a SUCCESS/FAILED bar chart\n",
"# - a horizontal bar chart of the top five IP addresses by failed attempts\n",
"# TODO 15: add titles and labels, save to output/auth_log_analysis.png, and call plt.show()\n"
]
},
{
"cell_type": "markdown",
"id": "fe92f571",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 5. Export results and write a short report\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "608ca490",
"metadata": {
"tags": [
"exercise"
]
},
"outputs": [],
"source": [
"# TODO 16: export clean_df and suspicious_ips to CSV\n",
"# TODO 17: create a summary string containing clean event count, FAILED count,\n",
"# failure rate, top IP, off-hours failures, and recommendations.\n",
"# TODO 18: save summary to output/capstone_summary.txt\n"
]
},
{
"cell_type": "markdown",
"id": "a6f05e23",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "10601dd1",
"metadata": {},
"outputs": [],
"source": [
"assert len(raw_df) > len(clean_df)\n",
"assert clean_df.duplicated().sum() == 0\n",
"assert failed_by_ip.index[0] == \"198.51.100.14\"\n",
"assert top_ip_count >= THRESHOLD\n",
"assert len(off_hours_failed) >= 1\n",
"assert (OUTPUT_DIR / \"auth_log_analysis.png\").exists()\n",
"assert (OUTPUT_DIR / \"capstone_summary.txt\").exists()\n",
"print(\"✓ Capstone lab completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "b56e4f9a",
"metadata": {},
"source": [
"## Report questions\n",
"\n",
"1. Which IP address should be investigated first, and what evidence supports that decision?\n",
"2. In what situations could a threshold of three failed attempts produce a false positive?\n",
"3. Which additional log sources should be reviewed before deciding to block an IP address?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Practical/data/auth_log.csv
+23
View File
@@ -0,0 +1,23 @@
timestamp,source_ip,username,status
2026-09-01T10:00:12Z,203.0.113.55,admin,SUCCESS
2026-09-01T10:01:23Z,198.51.100.14,guest,FAILED
2026-09-01T10:01:28Z,198.51.100.14,root,FAILED
2026-09-01T10:01:33Z,198.51.100.14,admin,FAILED
2026-09-01T10:02:15Z,203.0.113.55,jdoe,SUCCESS
2026-09-01T10:03:01Z,192.0.2.88,admin,SUCCESS
2026-09-01T10:03:45Z,198.51.100.14,jdoe,FAILED
2026-09-01T10:03:51Z,198.51.100.14,admin,FAILED
2026-09-01T10:03:51Z,198.51.100.14,admin,FAILED
2026-09-01T10:04:10Z,198.51.100.14,Admin, failed
bad-timestamp,198.51.100.14,root,FAILED
,203.0.113.55,backup,SUCCESS
2026-09-01T22:15:00Z,192.0.2.99,nightops,FAILED
2026-09-01T23:40:00Z,192.0.2.99,admin,FAILED
2026-09-02T01:05:00Z,198.51.100.14,ROOT,FAILED
2026-09-02T01:10:00Z,203.0.113.77,,FAILED
2026-09-02T06:10:00Z,203.0.113.55,alice,SUCCESS
2026-09-02T07:20:00Z,192.0.2.88,bob,SUCCESS
2026-09-02T07:25:00Z,192.0.2.88,bob,success
2026-09-02T09:00:00Z,,unknown,FAILED
2026-09-02T09:15:00Z,203.0.113.77,carol,LOCKED
2026-09-02T10:30:00Z,198.51.100.14,admin,FAILED
1 timestamp source_ip username status
2 2026-09-01T10:00:12Z 203.0.113.55 admin SUCCESS
3 2026-09-01T10:01:23Z 198.51.100.14 guest FAILED
4 2026-09-01T10:01:28Z 198.51.100.14 root FAILED
5 2026-09-01T10:01:33Z 198.51.100.14 admin FAILED
6 2026-09-01T10:02:15Z 203.0.113.55 jdoe SUCCESS
7 2026-09-01T10:03:01Z 192.0.2.88 admin SUCCESS
8 2026-09-01T10:03:45Z 198.51.100.14 jdoe FAILED
9 2026-09-01T10:03:51Z 198.51.100.14 admin FAILED
10 2026-09-01T10:03:51Z 198.51.100.14 admin FAILED
11 2026-09-01T10:04:10Z 198.51.100.14 Admin failed
12 bad-timestamp 198.51.100.14 root FAILED
13 203.0.113.55 backup SUCCESS
14 2026-09-01T22:15:00Z 192.0.2.99 nightops FAILED
15 2026-09-01T23:40:00Z 192.0.2.99 admin FAILED
16 2026-09-02T01:05:00Z 198.51.100.14 ROOT FAILED
17 2026-09-02T01:10:00Z 203.0.113.77 FAILED
18 2026-09-02T06:10:00Z 203.0.113.55 alice SUCCESS
19 2026-09-02T07:20:00Z 192.0.2.88 bob SUCCESS
20 2026-09-02T07:25:00Z 192.0.2.88 bob success
21 2026-09-02T09:00:00Z unknown FAILED
22 2026-09-02T09:15:00Z 203.0.113.77 carol LOCKED
23 2026-09-02T10:30:00Z 198.51.100.14 admin FAILED
Lab-2/Lab-2-Solution.zip
BIN
View File
Binary file not shown.
Lab-2/Lab-2-Solution/01_Python_Basics_Solution.ipynb
+233
View File
@@ -0,0 +1,233 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a9ae0aa5",
"metadata": {},
"source": [
"# Module 2 — Lab 1: Python Fundamentals for Security Analysis\n",
"\n",
"**Topics:** variables, data types, strings, type casting, operators\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "6b669205",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- store security information in variables and select suitable data types;\n",
"- normalize log strings using string methods;\n",
"- cast values read from logs to the required data types;\n",
"- combine comparison and logical operators to flag suspicious activity.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5338112e",
"metadata": {},
"outputs": [],
"source": [
"print(\"Hello, Cyber World!\")\n",
"\n",
"ip_address = \"198.51.100.14\"\n",
"port_number = 443\n",
"failed_logins = 7\n",
"response_time = 0.125\n",
"is_admin = False\n",
"\n",
"print(f\"IP: {ip_address} | Port: {port_number}\")\n",
"print(type(ip_address), type(port_number), type(response_time), type(is_admin))\n"
]
},
{
"cell_type": "markdown",
"id": "0094841a",
"metadata": {},
"source": [
"## 1. Strings in security logs\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b5ed12a4",
"metadata": {},
"outputs": [],
"source": [
"log_message = \" FAILED login for Admin from 198.51.100.14 \"\n",
"url = \"https://example.com/login\"\n",
"\n",
"print(log_message.strip())\n",
"print(log_message.lower())\n",
"print(log_message.replace(\"FAILED\", \"DENIED\"))\n",
"print(url[0:5])\n"
]
},
{
"cell_type": "markdown",
"id": "c484946c",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Normalize a username and extract the protocol\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c62e36d5",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"raw_username = \" AdMin \"\n",
"normalized_username = raw_username.strip().lower()\n",
"\n",
"url = \"https://example.com/login\"\n",
"protocol = url[0:5]\n",
"\n",
"print(normalized_username)\n",
"print(protocol)\n"
]
},
{
"cell_type": "markdown",
"id": "9f02318b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Type casting and operators\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61827b28",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"port_text = \"8080\"\n",
"port_number = int(port_text)\n",
"next_port = port_number + 1\n",
"\n",
"login_attempts = 7\n",
"user_ip = \"198.51.100.14\"\n",
"is_suspicious = login_attempts > 5 and user_ip != \"127.0.0.1\"\n",
"\n",
"print(f\"Next port: {next_port}\")\n",
"print(f\"Suspicious: {is_suspicious}\")\n"
]
},
{
"cell_type": "markdown",
"id": "0613c68c",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Calculate a basic risk score\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fa8d9c4d",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"failed_logins = 7\n",
"is_admin = False\n",
"is_off_hours = True\n",
"\n",
"risk_score = failed_logins * 10 + int(is_admin) * 25 + int(is_off_hours) * 15\n",
"requires_review = risk_score >= 60\n",
"\n",
"print(f\"Risk score: {risk_score}\")\n",
"print(f\"Requires review: {requires_review}\")\n"
]
},
{
"cell_type": "markdown",
"id": "285b59c6",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a59fbe2",
"metadata": {},
"outputs": [],
"source": [
"assert normalized_username == \"admin\"\n",
"assert protocol == \"https\"\n",
"assert next_port == 8081\n",
"assert is_suspicious is True\n",
"assert risk_score == 85\n",
"assert requires_review is True\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "92257020",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why does `port_text + 1` fail while `int(port_text) + 1` works?\n",
"2. What changes if you replace `and` with `or` in the `is_suspicious` condition?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/02_Control_Flow_Functions_Solution.ipynb
+242
View File
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "db19a07f",
"metadata": {},
"source": [
"# Module 2 — Lab 2: Control Flow and Functions\n",
" \n",
"**Topics:** if/elif/else, for, while, break, continue, function\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "0d178b19",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"You need to build small rules that classify login alerts and automatically review a list of IP addresses. All activity is simulated with local data; this notebook does not connect to or scan any real system.\n"
]
},
{
"cell_type": "markdown",
"id": "56bc45d4",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Classify failed login attempts\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dac7be01",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"failed_attempts = 7\n",
"\n",
"if failed_attempts >= 10:\n",
" severity = \"CRITICAL\"\n",
"elif failed_attempts >= 5:\n",
" severity = \"HIGH\"\n",
"elif failed_attempts >= 1:\n",
" severity = \"MEDIUM\"\n",
"else:\n",
" severity = \"LOW\"\n",
"\n",
"print(severity)\n"
]
},
{
"cell_type": "markdown",
"id": "63b6d1a8",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Review IP addresses with `for`, `continue`, and `break`\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3820130f",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"blocked_ips = {\"10.0.0.5\", \"10.0.0.12\"}\n",
"targets = [f\"10.0.0.{i}\" for i in range(1, 16)]\n",
"reviewed = []\n",
"\n",
"for ip in targets:\n",
" last_octet = int(ip.split(\".\")[-1])\n",
"\n",
" if last_octet % 2 != 0:\n",
" continue\n",
"\n",
" reviewed.append(ip)\n",
" print(f\"Reviewing: {ip}\")\n",
"\n",
" if ip in blocked_ips:\n",
" print(f\"Alert: blocked IP found: {ip}\")\n",
" break\n",
"\n",
"print(\"Reviewed:\", reviewed)\n"
]
},
{
"cell_type": "markdown",
"id": "61f09e15",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Limit retry attempts with `while`\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0ec1dc95",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"attempt = 0\n",
"max_attempts = 3\n",
"service_responded = False\n",
"\n",
"while attempt < max_attempts and not service_responded:\n",
" attempt += 1\n",
" print(f\"Health check attempt {attempt}\")\n",
" service_responded = attempt == 2\n",
"\n",
"print(f\"Service responded: {service_responded}\")\n"
]
},
{
"cell_type": "markdown",
"id": "b849e734",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Package a rule in a function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a34324a1",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"def classify_login_risk(failed_attempts, is_admin=False, is_off_hours=False):\n",
" # Return LOW, MEDIUM, HIGH, or CRITICAL for a login event.\n",
" score = failed_attempts * 10\n",
" if is_admin:\n",
" score += 25\n",
" if is_off_hours:\n",
" score += 15\n",
"\n",
" if score >= 100:\n",
" return \"CRITICAL\"\n",
" if score >= 60:\n",
" return \"HIGH\"\n",
" if score >= 30:\n",
" return \"MEDIUM\"\n",
" return \"LOW\"\n",
"\n",
"\n",
"print(classify_login_risk(2))\n",
"print(classify_login_risk(7, is_off_hours=True))\n",
"print(classify_login_risk(8, is_admin=True))\n"
]
},
{
"cell_type": "markdown",
"id": "aba3b1a8",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de52830d",
"metadata": {},
"outputs": [],
"source": [
"assert severity == \"HIGH\"\n",
"assert reviewed == [\"10.0.0.2\", \"10.0.0.4\", \"10.0.0.6\", \"10.0.0.8\", \"10.0.0.10\", \"10.0.0.12\"]\n",
"assert attempt == 2 and service_responded is True\n",
"assert classify_login_risk(2) == \"LOW\"\n",
"assert classify_login_risk(7, is_off_hours=True) == \"HIGH\"\n",
"assert classify_login_risk(8, is_admin=True) == \"CRITICAL\"\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "c97e0f71",
"metadata": {},
"source": [
"## Extension challenge\n",
"\n",
"Modify the function to add 20 points when `source_ip` belongs to a set of blocked IP addresses. Write at least two `assert` statements for the new rule.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/03_Data_Structures_Log_Parsing_Solution.ipynb
+242
View File
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "06f7188c",
"metadata": {},
"source": [
"# Module 2 — Lab 3: Data Structures and Raw Log Analysis\n",
" \n",
"**Topics:** list, tuple, dictionary, set, parsing\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "67ad00e0",
"metadata": {},
"source": [
"## Input data\n",
"\n",
"Each log line contains four comma-separated fields: `timestamp,source_ip,username,status`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4b49fc55",
"metadata": {},
"outputs": [],
"source": [
"raw_logs = [\n",
" \"2026-09-01T08:00:00Z,203.0.113.55,alice,SUCCESS\",\n",
" \"2026-09-01T08:01:00Z,198.51.100.14,admin,FAILED\",\n",
" \"2026-09-01T08:01:20Z,198.51.100.14,root,FAILED\",\n",
" \"2026-09-01T08:02:00Z,192.0.2.88,bob,SUCCESS\",\n",
" \"2026-09-01T08:02:30Z,198.51.100.14,guest,FAILED\",\n",
" \"2026-09-01T08:03:00Z,192.0.2.99,carol,FAILED\",\n",
"]\n",
"\n",
"critical_fields = (\"timestamp\", \"source_ip\", \"username\", \"status\")\n",
"blocked_ips = {\"198.51.100.14\", \"203.0.113.200\"}\n",
"\n",
"print(f\"Rows: {len(raw_logs)}\")\n",
"print(\"Critical fields:\", critical_fields)\n"
]
},
{
"cell_type": "markdown",
"id": "3f8fcb17",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Convert one log line into a dictionary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47d1eb8b",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"def parse_log_line(line):\n",
" timestamp, source_ip, username, status = line.split(\",\")\n",
" return {\n",
" \"timestamp\": timestamp,\n",
" \"source_ip\": source_ip,\n",
" \"username\": username.lower(),\n",
" \"status\": status.upper(),\n",
" }\n",
"\n",
"\n",
"sample_event = parse_log_line(raw_logs[0])\n",
"print(sample_event)\n"
]
},
{
"cell_type": "markdown",
"id": "56f0258d",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Create an event list and a set of unique IP addresses\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a8c8af5e",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"events = [parse_log_line(line) for line in raw_logs]\n",
"unique_ips = {event[\"source_ip\"] for event in events}\n",
"\n",
"print(\"Unique IPs:\", sorted(unique_ips))\n",
"print(\"Number of unique IPs:\", len(unique_ips))\n"
]
},
{
"cell_type": "markdown",
"id": "7651a31c",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Count failed logins with a dictionary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c0a3675",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"failed_by_ip = {}\n",
"\n",
"for event in events:\n",
" if event[\"status\"] != \"FAILED\":\n",
" continue\n",
" ip = event[\"source_ip\"]\n",
" failed_by_ip[ip] = failed_by_ip.get(ip, 0) + 1\n",
"\n",
"print(failed_by_ip)\n"
]
},
{
"cell_type": "markdown",
"id": "21436c12",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Combine dictionaries and sets to create alerts\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11954df9",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"alerts = []\n",
"\n",
"for ip, count in failed_by_ip.items():\n",
" if count >= 3 or ip in blocked_ips:\n",
" alerts.append({\n",
" \"source_ip\": ip,\n",
" \"failed_attempts\": count,\n",
" \"blocked\": ip in blocked_ips,\n",
" })\n",
"\n",
"print(alerts)\n"
]
},
{
"cell_type": "markdown",
"id": "b18d5731",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d30adfba",
"metadata": {},
"outputs": [],
"source": [
"assert sample_event[\"username\"] == \"alice\"\n",
"assert len(events) == 6\n",
"assert unique_ips == {\"203.0.113.55\", \"198.51.100.14\", \"192.0.2.88\", \"192.0.2.99\"}\n",
"assert failed_by_ip[\"198.51.100.14\"] == 3\n",
"assert alerts == [{\"source_ip\": \"198.51.100.14\", \"failed_attempts\": 3, \"blocked\": True}]\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "55d97609",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"- Why is a `set` suitable for storing unique IP addresses or a blocklist?\n",
"- Why is a `tuple` suitable for field names that should not change?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/04_NumPy_Security_Metrics_Solution.ipynb
+200
View File
@@ -0,0 +1,200 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "03d77301",
"metadata": {},
"source": [
"# Module 2 — Lab 4: NumPy for Security Metrics\n",
"\n",
"\n",
"**Topics:** array, vectorization, statistics, boolean mask\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "33cb0ba1",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"A network sensor reports packet sizes, latency, and failed authentication attempts for each time window. We use NumPy to calculate efficiently across the entire array.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d7d3de72",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"packet_sizes = np.array([64, 128, 1500, 512, 1514, 256, 1400, 80])\n",
"latency_ms = np.array([12.5, 18.0, 250.0, 33.5, 180.0, 25.0, 95.0, 10.0])\n",
"failed_attempts = np.array([0, 1, 8, 2, 10, 1, 6, 0])\n",
"\n",
"print(packet_sizes.shape, packet_sizes.dtype)\n"
]
},
{
"cell_type": "markdown",
"id": "5d225bd1",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Unit conversion and descriptive statistics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3da57f45",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"packet_kb = packet_sizes / 1024\n",
"mean_latency = latency_ms.mean()\n",
"p95_latency = np.percentile(latency_ms, 95)\n",
"\n",
"print(\"Packet sizes (KB):\", np.round(packet_kb, 3))\n",
"print(f\"Mean latency: {mean_latency:.2f} ms\")\n",
"print(f\"P95 latency: {p95_latency:.2f} ms\")\n"
]
},
{
"cell_type": "markdown",
"id": "dddbd812",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Filter with a Boolean mask\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d24785c0",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"suspicious_mask = (packet_sizes > 1400) | (latency_ms > 150) | (failed_attempts >= 8)\n",
"suspicious_indices = np.where(suspicious_mask)[0]\n",
"\n",
"print(\"Mask:\", suspicious_mask)\n",
"print(\"Suspicious indices:\", suspicious_indices)\n",
"print(\"Suspicious packet sizes:\", packet_sizes[suspicious_mask])\n"
]
},
{
"cell_type": "markdown",
"id": "ce903e91",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Normalize values and create a risk score\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcece1eb",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"def min_max(values):\n",
" values = np.asarray(values, dtype=float)\n",
" value_range = values.max() - values.min()\n",
" if value_range == 0:\n",
" return np.zeros_like(values)\n",
" return (values - values.min()) / value_range\n",
"\n",
"\n",
"latency_score = min_max(latency_ms)\n",
"failure_score = min_max(failed_attempts)\n",
"risk_score = 0.4 * latency_score + 0.6 * failure_score\n",
"\n",
"print(\"Risk scores:\", np.round(risk_score, 3))\n",
"print(\"Highest-risk index:\", int(np.argmax(risk_score)))\n"
]
},
{
"cell_type": "markdown",
"id": "65a088f0",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a730cc89",
"metadata": {},
"outputs": [],
"source": [
"assert packet_sizes.shape == (8,)\n",
"assert np.isclose(mean_latency, 78.0)\n",
"assert suspicious_indices.tolist() == [2, 4]\n",
"assert int(np.argmax(risk_score)) == 4\n",
"assert np.all((risk_score >= 0) & (risk_score <= 1))\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "2c85c6e5",
"metadata": {},
"source": [
"## Extension challenge\n",
"\n",
"Change the weights assigned to latency and failed attempts. Observe whether the highest-risk element changes and explain why.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/05_Pandas_Log_Cleaning_Solution.ipynb
+256
View File
@@ -0,0 +1,256 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "97d2f264",
"metadata": {},
"source": [
"# Module 2 — Lab 5: Pandas — Loading, Inspecting, and Cleaning Logs\n",
"\n",
"\n",
"**Topics:** CSV, DataFrame, missing values, duplicates, datetime, filtering\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "7b1ce82e",
"metadata": {},
"source": [
"## Scenario\n",
"\n",
"The `data/auth_log.csv` file contains duplicate rows, invalid timestamps, missing critical fields, and inconsistent capitalization. The goal is to produce a clean DataFrame ready for analysis.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9dea9739",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/auth_log.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"\n",
"log_df = pd.read_csv(DATA_PATH)\n",
"log_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "6e6da8ed",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Inspect data quality\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9ef821c",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"print(\"Shape:\", log_df.shape)\n",
"print(\"Duplicate rows:\", log_df.duplicated().sum())\n",
"print(\"Missing values by column:\")\n",
"print(log_df.isna().sum())\n",
"print(\"\\nData types:\")\n",
"print(log_df.dtypes)\n"
]
},
{
"cell_type": "markdown",
"id": "6b759b1c",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Remove duplicates and normalize timestamps and critical fields\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6d4342ee",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"clean_df = log_df.copy()\n",
"clean_df = clean_df.drop_duplicates()\n",
"\n",
"clean_df[\"timestamp\"] = pd.to_datetime(\n",
" clean_df[\"timestamp\"], errors=\"coerce\", utc=True\n",
")\n",
"\n",
"clean_df = clean_df.dropna(\n",
" subset=[\"timestamp\", \"source_ip\", \"status\"]\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "c5c59040",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Normalize text and create time-based fields\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "012ba902",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"clean_df[\"source_ip\"] = clean_df[\"source_ip\"].str.strip()\n",
"clean_df[\"username\"] = clean_df[\"username\"].fillna(\"unknown\").str.strip().str.lower()\n",
"clean_df[\"status\"] = clean_df[\"status\"].str.strip().str.upper()\n",
"\n",
"clean_df = clean_df[clean_df[\"status\"].isin([\"SUCCESS\", \"FAILED\"])]\n",
"clean_df[\"hour\"] = clean_df[\"timestamp\"].dt.hour\n",
"clean_df = clean_df.sort_values(\"timestamp\").reset_index(drop=True)\n",
"\n",
"clean_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "354fc8c0",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Filter and aggregate\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "38f69103",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"failed_df = clean_df[clean_df[\"status\"] == \"FAILED\"].copy()\n",
"failed_by_ip = failed_df[\"source_ip\"].value_counts().rename(\"failed_attempts\")\n",
"status_counts = clean_df[\"status\"].value_counts()\n",
"\n",
"print(\"Status counts:\")\n",
"print(status_counts)\n",
"print(\"\\nFailed attempts by IP:\")\n",
"print(failed_by_ip)\n"
]
},
{
"cell_type": "markdown",
"id": "748f22a9",
"metadata": {},
"source": [
"## 5. Export clean data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fdcb2bf7",
"metadata": {},
"outputs": [],
"source": [
"clean_df.to_csv(OUTPUT_DIR / \"clean_auth_log.csv\", index=False)\n",
"clean_df.to_json(\n",
" OUTPUT_DIR / \"clean_auth_log.json\",\n",
" orient=\"records\",\n",
" indent=2,\n",
" date_format=\"iso\",\n",
")\n",
"failed_by_ip.to_csv(\n",
" OUTPUT_DIR / \"failed_login_by_ip.csv\",\n",
" header=True,\n",
")\n",
"\n",
"print(\"Exported files to:\", OUTPUT_DIR.resolve())\n"
]
},
{
"cell_type": "markdown",
"id": "4b07b67e",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b16f153c",
"metadata": {},
"outputs": [],
"source": [
"assert log_df.duplicated().sum() >= 1\n",
"assert clean_df.duplicated().sum() == 0\n",
"assert clean_df[\"timestamp\"].isna().sum() == 0\n",
"assert set(clean_df[\"status\"]) <= {\"SUCCESS\", \"FAILED\"}\n",
"assert clean_df[\"username\"].str.strip().eq(clean_df[\"username\"]).all()\n",
"assert failed_by_ip.index[0] == \"198.51.100.14\"\n",
"assert (OUTPUT_DIR / \"clean_auth_log.csv\").exists()\n",
"assert (OUTPUT_DIR / \"clean_auth_log.json\").exists()\n",
"print(\"✓ Lab 5 completed\")\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/06_Capstone_Auth_Log_Analysis_Solution.ipynb
+325
View File
@@ -0,0 +1,325 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1727969b",
"metadata": {},
"source": [
"# Module 2 — Lab 6: Capstone — Investigating Suspicious Logins\n",
"\n",
" \n",
"**Topics:** Pandas, NumPy, Matplotlib, export, analyst summary\n",
"\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "467cdb20",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"As a junior security analyst, you must build a complete workflow:\n",
"\n",
"1. load and clean authentication logs;\n",
"2. identify IP addresses with repeated failed logins;\n",
"3. identify off-hours activity;\n",
"4. visualize the results;\n",
"5. export the data and write a short conclusion.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "525cb519",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/auth_log.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n"
]
},
{
"cell_type": "markdown",
"id": "d94da21b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 1. Write a reusable cleaning function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c33bbbcb",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"def clean_auth_log(df):\n",
" result = df.copy().drop_duplicates()\n",
" result[\"timestamp\"] = pd.to_datetime(\n",
" result[\"timestamp\"], errors=\"coerce\", utc=True\n",
" )\n",
" result = result.dropna(subset=[\"timestamp\", \"source_ip\", \"status\"])\n",
" result[\"source_ip\"] = result[\"source_ip\"].str.strip()\n",
" result[\"username\"] = result[\"username\"].fillna(\"unknown\").str.strip().str.lower()\n",
" result[\"status\"] = result[\"status\"].str.strip().str.upper()\n",
" result = result[result[\"status\"].isin([\"SUCCESS\", \"FAILED\"])]\n",
" result[\"hour\"] = result[\"timestamp\"].dt.hour\n",
" result[\"is_off_hours\"] = (result[\"hour\"] < 6) | (result[\"hour\"] >= 22)\n",
" return result.sort_values(\"timestamp\").reset_index(drop=True)\n",
"\n",
"\n",
"raw_df = pd.read_csv(DATA_PATH)\n",
"clean_df = clean_auth_log(raw_df)\n",
"print(f\"Raw rows: {len(raw_df)} | Clean rows: {len(clean_df)}\")\n",
"clean_df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "5205cafa",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 2. Analyze failures and rank IP addresses\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "085dd483",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"failed_df = clean_df[clean_df[\"status\"] == \"FAILED\"].copy()\n",
"failed_by_ip = (\n",
" failed_df.groupby(\"source_ip\")\n",
" .size()\n",
" .sort_values(ascending=False)\n",
" .rename(\"failed_attempts\")\n",
")\n",
"\n",
"THRESHOLD = 3\n",
"suspicious_ips = failed_by_ip[failed_by_ip >= THRESHOLD]\n",
"\n",
"print(\"Failed attempts by IP:\")\n",
"print(failed_by_ip)\n",
"print(\"\\nSuspicious IPs:\")\n",
"print(suspicious_ips)\n"
]
},
{
"cell_type": "markdown",
"id": "5967176b",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 3. Calculate additional investigation metrics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7cab677c",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"status_counts = clean_df[\"status\"].value_counts().reindex([\"SUCCESS\", \"FAILED\"], fill_value=0)\n",
"off_hours_failed = failed_df[failed_df[\"is_off_hours\"]]\n",
"top_ip = failed_by_ip.index[0]\n",
"top_ip_count = int(failed_by_ip.iloc[0])\n",
"failure_rate = np.round((status_counts[\"FAILED\"] / status_counts.sum()) * 100, 2)\n",
"\n",
"print(f\"Top suspicious IP: {top_ip} ({top_ip_count} failures)\")\n",
"print(f\"Failure rate: {failure_rate}%\")\n",
"print(f\"Off-hours failed attempts: {len(off_hours_failed)}\")\n"
]
},
{
"cell_type": "markdown",
"id": "c472f43a",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 4. Visualize the findings\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4816f960",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"fig, axes = plt.subplots(1, 2, figsize=(11, 4))\n",
"\n",
"status_counts.plot(\n",
" kind=\"bar\",\n",
" ax=axes[0],\n",
" color=[\"#2a9d8f\", \"#e76f51\"],\n",
" title=\"Authentication status\",\n",
")\n",
"axes[0].set_xlabel(\"Status\")\n",
"axes[0].set_ylabel(\"Number of attempts\")\n",
"axes[0].tick_params(axis=\"x\", rotation=0)\n",
"\n",
"failed_by_ip.head(5).sort_values().plot(\n",
" kind=\"barh\",\n",
" ax=axes[1],\n",
" color=\"#e76f51\",\n",
" title=\"Top IPs by failed logins\",\n",
")\n",
"axes[1].set_xlabel(\"Failed attempts\")\n",
"axes[1].set_ylabel(\"Source IP\")\n",
"\n",
"fig.suptitle(\"Authentication Log Investigation\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"auth_log_analysis.png\", dpi=150, bbox_inches=\"tight\")\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"id": "933a5898",
"metadata": {
"tags": [
"exercise"
]
},
"source": [
"## 5. Export results and write a short report\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc68ed13",
"metadata": {
"tags": [
"solution"
]
},
"outputs": [],
"source": [
"clean_df.to_csv(OUTPUT_DIR / \"capstone_clean_auth_log.csv\", index=False)\n",
"suspicious_ips.to_csv(\n",
" OUTPUT_DIR / \"capstone_suspicious_ips.csv\",\n",
" header=True,\n",
")\n",
"\n",
"summary = f'''SECURITY ANALYSIS SUMMARY\n",
"=========================\n",
"Clean events: {len(clean_df)}\n",
"Failed events: {int(status_counts['FAILED'])}\n",
"Failure rate: {failure_rate}%\n",
"Top suspicious IP: {top_ip} ({top_ip_count} failed attempts)\n",
"Off-hours failed attempts: {len(off_hours_failed)}\n",
"Decision threshold: {THRESHOLD} failed attempts\n",
"\n",
"Recommended action:\n",
"- Review events from {top_ip}.\n",
"- Correlate off-hours failures with endpoint and identity logs.\n",
"- Validate whether the account owners recognize these attempts.\n",
"'''\n",
"\n",
"(OUTPUT_DIR / \"capstone_summary.txt\").write_text(summary, encoding=\"utf-8\")\n",
"print(summary)\n"
]
},
{
"cell_type": "markdown",
"id": "023ea53a",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "560fa29e",
"metadata": {},
"outputs": [],
"source": [
"assert len(raw_df) > len(clean_df)\n",
"assert clean_df.duplicated().sum() == 0\n",
"assert failed_by_ip.index[0] == \"198.51.100.14\"\n",
"assert top_ip_count >= THRESHOLD\n",
"assert len(off_hours_failed) >= 1\n",
"assert (OUTPUT_DIR / \"auth_log_analysis.png\").exists()\n",
"assert (OUTPUT_DIR / \"capstone_summary.txt\").exists()\n",
"print(\"✓ Capstone lab completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "dedf0889",
"metadata": {},
"source": [
"## Report questions\n",
"\n",
"1. Which IP address should be investigated first, and what evidence supports that decision?\n",
"2. In what situations could a threshold of three failed attempts produce a false positive?\n",
"3. Which additional log sources should be reviewed before deciding to block an IP address?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-2/Lab-2-Solution/data/auth_log.csv
+23
View File
@@ -0,0 +1,23 @@
timestamp,source_ip,username,status
2026-09-01T10:00:12Z,203.0.113.55,admin,SUCCESS
2026-09-01T10:01:23Z,198.51.100.14,guest,FAILED
2026-09-01T10:01:28Z,198.51.100.14,root,FAILED
2026-09-01T10:01:33Z,198.51.100.14,admin,FAILED
2026-09-01T10:02:15Z,203.0.113.55,jdoe,SUCCESS
2026-09-01T10:03:01Z,192.0.2.88,admin,SUCCESS
2026-09-01T10:03:45Z,198.51.100.14,jdoe,FAILED
2026-09-01T10:03:51Z,198.51.100.14,admin,FAILED
2026-09-01T10:03:51Z,198.51.100.14,admin,FAILED
2026-09-01T10:04:10Z,198.51.100.14,Admin, failed
bad-timestamp,198.51.100.14,root,FAILED
,203.0.113.55,backup,SUCCESS
2026-09-01T22:15:00Z,192.0.2.99,nightops,FAILED
2026-09-01T23:40:00Z,192.0.2.99,admin,FAILED
2026-09-02T01:05:00Z,198.51.100.14,ROOT,FAILED
2026-09-02T01:10:00Z,203.0.113.77,,FAILED
2026-09-02T06:10:00Z,203.0.113.55,alice,SUCCESS
2026-09-02T07:20:00Z,192.0.2.88,bob,SUCCESS
2026-09-02T07:25:00Z,192.0.2.88,bob,success
2026-09-02T09:00:00Z,,unknown,FAILED
2026-09-02T09:15:00Z,203.0.113.77,carol,LOCKED
2026-09-02T10:30:00Z,198.51.100.14,admin,FAILED
1 timestamp source_ip username status
2 2026-09-01T10:00:12Z 203.0.113.55 admin SUCCESS
3 2026-09-01T10:01:23Z 198.51.100.14 guest FAILED
4 2026-09-01T10:01:28Z 198.51.100.14 root FAILED
5 2026-09-01T10:01:33Z 198.51.100.14 admin FAILED
6 2026-09-01T10:02:15Z 203.0.113.55 jdoe SUCCESS
7 2026-09-01T10:03:01Z 192.0.2.88 admin SUCCESS
8 2026-09-01T10:03:45Z 198.51.100.14 jdoe FAILED
9 2026-09-01T10:03:51Z 198.51.100.14 admin FAILED
10 2026-09-01T10:03:51Z 198.51.100.14 admin FAILED
11 2026-09-01T10:04:10Z 198.51.100.14 Admin failed
12 bad-timestamp 198.51.100.14 root FAILED
13 203.0.113.55 backup SUCCESS
14 2026-09-01T22:15:00Z 192.0.2.99 nightops FAILED
15 2026-09-01T23:40:00Z 192.0.2.99 admin FAILED
16 2026-09-02T01:05:00Z 198.51.100.14 ROOT FAILED
17 2026-09-02T01:10:00Z 203.0.113.77 FAILED
18 2026-09-02T06:10:00Z 203.0.113.55 alice SUCCESS
19 2026-09-02T07:20:00Z 192.0.2.88 bob SUCCESS
20 2026-09-02T07:25:00Z 192.0.2.88 bob success
21 2026-09-02T09:00:00Z unknown FAILED
22 2026-09-02T09:15:00Z 203.0.113.77 carol LOCKED
23 2026-09-02T10:30:00Z 198.51.100.14 admin FAILED
Lab-2/Lab-2-Solution/output/auth_log_analysis.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

Lab-2/Lab-2-Solution/output/capstone_clean_auth_log.csv
+18
View File
@@ -0,0 +1,18 @@
timestamp,source_ip,username,status,hour,is_off_hours
2026-09-01 10:00:12+00:00,203.0.113.55,admin,SUCCESS,10,False
2026-09-01 10:01:23+00:00,198.51.100.14,guest,FAILED,10,False
2026-09-01 10:01:28+00:00,198.51.100.14,root,FAILED,10,False
2026-09-01 10:01:33+00:00,198.51.100.14,admin,FAILED,10,False
2026-09-01 10:02:15+00:00,203.0.113.55,jdoe,SUCCESS,10,False
2026-09-01 10:03:01+00:00,192.0.2.88,admin,SUCCESS,10,False
2026-09-01 10:03:45+00:00,198.51.100.14,jdoe,FAILED,10,False
2026-09-01 10:03:51+00:00,198.51.100.14,admin,FAILED,10,False
2026-09-01 10:04:10+00:00,198.51.100.14,admin,FAILED,10,False
2026-09-01 22:15:00+00:00,192.0.2.99,nightops,FAILED,22,True
2026-09-01 23:40:00+00:00,192.0.2.99,admin,FAILED,23,True
2026-09-02 01:05:00+00:00,198.51.100.14,root,FAILED,1,True
2026-09-02 01:10:00+00:00,203.0.113.77,unknown,FAILED,1,True
2026-09-02 06:10:00+00:00,203.0.113.55,alice,SUCCESS,6,False
2026-09-02 07:20:00+00:00,192.0.2.88,bob,SUCCESS,7,False
2026-09-02 07:25:00+00:00,192.0.2.88,bob,SUCCESS,7,False
2026-09-02 10:30:00+00:00,198.51.100.14,admin,FAILED,10,False
1 timestamp source_ip username status hour is_off_hours
2 2026-09-01 10:00:12+00:00 203.0.113.55 admin SUCCESS 10 False
3 2026-09-01 10:01:23+00:00 198.51.100.14 guest FAILED 10 False
4 2026-09-01 10:01:28+00:00 198.51.100.14 root FAILED 10 False
5 2026-09-01 10:01:33+00:00 198.51.100.14 admin FAILED 10 False
6 2026-09-01 10:02:15+00:00 203.0.113.55 jdoe SUCCESS 10 False
7 2026-09-01 10:03:01+00:00 192.0.2.88 admin SUCCESS 10 False
8 2026-09-01 10:03:45+00:00 198.51.100.14 jdoe FAILED 10 False
9 2026-09-01 10:03:51+00:00 198.51.100.14 admin FAILED 10 False
10 2026-09-01 10:04:10+00:00 198.51.100.14 admin FAILED 10 False
11 2026-09-01 22:15:00+00:00 192.0.2.99 nightops FAILED 22 True
12 2026-09-01 23:40:00+00:00 192.0.2.99 admin FAILED 23 True
13 2026-09-02 01:05:00+00:00 198.51.100.14 root FAILED 1 True
14 2026-09-02 01:10:00+00:00 203.0.113.77 unknown FAILED 1 True
15 2026-09-02 06:10:00+00:00 203.0.113.55 alice SUCCESS 6 False
16 2026-09-02 07:20:00+00:00 192.0.2.88 bob SUCCESS 7 False
17 2026-09-02 07:25:00+00:00 192.0.2.88 bob SUCCESS 7 False
18 2026-09-02 10:30:00+00:00 198.51.100.14 admin FAILED 10 False
Lab-2/Lab-2-Solution/output/capstone_summary.txt
+13
View File
@@ -0,0 +1,13 @@
SECURITY ANALYSIS SUMMARY
=========================
Clean events: 17
Failed events: 11
Failure rate: 64.71%
Top suspicious IP: 198.51.100.14 (8 failed attempts)
Off-hours failed attempts: 4
Decision threshold: 3 failed attempts
Recommended action:
- Review events from 198.51.100.14.
- Correlate off-hours failures with endpoint and identity logs.
- Validate whether the account owners recognize these attempts.
Lab-2/Lab-2-Solution/output/capstone_suspicious_ips.csv
+2
View File
@@ -0,0 +1,2 @@
source_ip,failed_attempts
198.51.100.14,8
1 source_ip failed_attempts
2 198.51.100.14 8
Lab-3/Lab-3-Practical.zip
BIN
View File
Binary file not shown.
Lab-3/Lab-3-Practical/01_Security_Features_Labels_Lab.ipynb
+204
View File
@@ -0,0 +1,204 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "578d1d03",
"metadata": {},
"source": [
"# Module 3 — Lab 1: Security Features and Binary Labels\n",
" \n",
"**Topics:** samples, features, labels, class distribution\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "a79a9968",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- identify samples, features, and labels in a security dataset;\n",
"- inspect the schema and data types of a network-flow table;\n",
"- convert multiple attack names into one binary target;\n",
"- explain why class distribution matters before model training.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7981983b",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
"RANDOM_STATE = 42\n"
]
},
{
"cell_type": "markdown",
"id": "7b199e34",
"metadata": {},
"source": [
"## 1. Load and inspect the flow table\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7a13f6b9",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load DATA_PATH into a DataFrame named df\n",
"df = None\n",
"\n",
"# TODO 2: display the first five rows\n",
"print(\"Shape:\", None)\n",
"df\n"
]
},
{
"cell_type": "markdown",
"id": "8eed67bf",
"metadata": {},
"source": [
"## 2. Identify the label and numeric features\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "509572c0",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: store the original label column in y_text\n",
"y_text = None\n",
"\n",
"# TODO 4: create X using only numeric columns and exclude Label\n",
"X = None\n",
"\n",
"print(\"Feature matrix shape:\", None)\n",
"print(\"Original labels:\")\n",
"print(None)\n"
]
},
{
"cell_type": "markdown",
"id": "a312a6cd",
"metadata": {},
"source": [
"## 3. Create a binary security target\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c24823de",
"metadata": {},
"outputs": [],
"source": [
"# BENIGN becomes 0. Every attack family becomes 1.\n",
"# TODO 5: normalize labels with strip() and upper(), then create y_binary\n",
"y_binary = None\n",
"\n",
"# TODO 6: calculate class_counts sorted by class index\n",
"class_counts = None\n",
"\n",
"print(class_counts)\n",
"print(\"Attack rate:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "123f407c",
"metadata": {},
"source": [
"## 4. Connect rows to feature vectors\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f85c4106",
"metadata": {},
"outputs": [],
"source": [
"sample_index = 0\n",
"\n",
"# TODO 7: select one row from X as sample_features and its target as sample_target\n",
"sample_features = None\n",
"sample_target = None\n",
"\n",
"print(\"Selected feature vector:\")\n",
"print(sample_features)\n",
"print(\"Binary target:\", sample_target)\n"
]
},
{
"cell_type": "markdown",
"id": "0d228da1",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "37f4f448",
"metadata": {},
"outputs": [],
"source": [
"assert isinstance(df, pd.DataFrame)\n",
"assert \"Label\" not in X.columns\n",
"assert set(y_binary.unique()) == {0, 1}\n",
"assert len(X) == len(y_binary)\n",
"assert class_counts.loc[0] > class_counts.loc[1]\n",
"assert len(sample_features) == X.shape[1]\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "499400fe",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is each row considered one sample while each column is considered one feature?\n",
"2. What information is lost when several attack families are mapped to one `ATTACK` class?\n",
"3. Why could high accuracy be misleading when `BENIGN` is the majority class?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/02_Clean_Split_No_Leakage_Lab.ipynb
+207
View File
@@ -0,0 +1,207 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d1c571ce",
"metadata": {},
"source": [
"# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
"\n",
"\n",
"**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "209fc458",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- remove duplicate records and normalize invalid numeric values;\n",
"- create a stratified train-test split;\n",
"- fit imputation and scaling operations on training data only;\n",
"- explain how fitting preprocessing on the test set causes leakage.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "660ee68c",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.preprocessing import StandardScaler\n"
]
},
{
"cell_type": "markdown",
"id": "d2a03355",
"metadata": {},
"source": [
"## 1. Load and clean the raw table\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8962439b",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(DATA_PATH)\n",
"\n",
"# TODO 1: remove surrounding whitespace from every column name\n",
"raw_df.columns = None\n",
"\n",
"# TODO 2: remove duplicate rows and store the result in clean_df\n",
"clean_df = None\n",
"\n",
"# TODO 3: replace positive and negative infinity with NaN\n",
"clean_df = None\n",
"\n",
"print(\"Raw rows:\", len(raw_df))\n",
"print(\"Clean rows:\", None)\n",
"print(\"Missing numeric values:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "840a6b05",
"metadata": {},
"source": [
"## 2. Create features, target, and a stratified split\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4bd85328",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: select numeric features and create a binary target\n",
"X = None\n",
"y = None\n",
"\n",
"# TODO 5: create an 80/20 split with random_state=42 and stratify=y\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"print(\"Train shape:\", None)\n",
"print(\"Test shape:\", None)\n",
"print(\"Train attack rate:\", None)\n",
"print(\"Test attack rate:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "0408dc06",
"metadata": {},
"source": [
"## 3. Fit preprocessing on training data only\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88ffbc3e",
"metadata": {},
"outputs": [],
"source": [
"imputer = SimpleImputer(strategy=\"median\")\n",
"scaler = StandardScaler()\n",
"\n",
"# TODO 6: fit the imputer on X_train, then transform train and test\n",
"X_train_imputed = None\n",
"X_test_imputed = None\n",
"\n",
"# TODO 7: fit the scaler on imputed training data, then transform train and test\n",
"X_train_scaled = None\n",
"X_test_scaled = None\n",
"\n",
"print(\"Mean of scaled training columns:\", None)\n",
"print(\"Mean of scaled test columns:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "578c5e87",
"metadata": {},
"source": [
"## Why the test mean is not exactly zero\n",
"\n",
"`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
]
},
{
"cell_type": "markdown",
"id": "639b3502",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b00a51c",
"metadata": {},
"outputs": [],
"source": [
"assert len(clean_df) < len(raw_df)\n",
"assert np.isfinite(X_train_scaled).all()\n",
"assert np.isfinite(X_test_scaled).all()\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
"assert X_train_scaled.shape[1] == X.shape[1]\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "e0461854",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must the imputer learn medians from the training set only?\n",
"2. What information would leak if the scaler were fit before the train-test split?\n",
"3. Why is stratification useful for an imbalanced security target?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/03_Random_Forest_Pipeline_Lab.ipynb
+216
View File
@@ -0,0 +1,216 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "2eb3034b",
"metadata": {},
"source": [
"# Module 3 — Lab 3: A Random Forest Security Pipeline\n",
"\n",
"**Topics:** Scikit-learn Pipeline, imputation, Random Forest, fit, predict\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "7bf6c5fd",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- create one reusable preprocessing-and-model pipeline;\n",
"- train a Random Forest binary classifier;\n",
"- generate predictions for unseen data;\n",
"- explain why tree models do not require feature standardization.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c98b721f",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n"
]
},
{
"cell_type": "markdown",
"id": "3888c4bb",
"metadata": {},
"source": [
"## 1. Prepare the binary classification data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4c28e44",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH)\n",
"df.columns = df.columns.str.strip()\n",
"df = df.drop_duplicates().replace([np.inf, -np.inf], np.nan)\n",
"\n",
"# TODO 1: create numeric X and binary y\n",
"X = None\n",
"y = None\n",
"\n",
"# TODO 2: create a stratified 80/20 split\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n"
]
},
{
"cell_type": "markdown",
"id": "6aad53d0",
"metadata": {},
"source": [
"## 2. Build one pipeline\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e6c00af0",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create a Pipeline with:\n",
"# - SimpleImputer(strategy=\"median\")\n",
"# - RandomForestClassifier with n_estimators=100, max_depth=12,\n",
"# random_state=42, and n_jobs=-1\n",
"pipeline = None\n",
"\n",
"print(pipeline)\n"
]
},
{
"cell_type": "markdown",
"id": "c2111199",
"metadata": {},
"source": [
"## Why there is no `StandardScaler`\n",
"\n",
"Random Forest makes decisions by comparing feature values with thresholds. Scaling a column does not change the ordering of its values, so standardization is not required for this model family. The imputer remains inside the pipeline because the model needs finite values.\n"
]
},
{
"cell_type": "markdown",
"id": "16aa1030",
"metadata": {},
"source": [
"## 3. Train and predict\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "59b48b8d",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: fit the complete pipeline on X_train and y_train\n",
"\n",
"# TODO 5: predict labels for X_test and store them in y_pred\n",
"y_pred = None\n",
"\n",
"print(\"Predictions:\", None)\n",
"print(\"Number of test samples:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "c0c83354",
"metadata": {},
"source": [
"## 4. Predict one flow\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1910ac4",
"metadata": {},
"outputs": [],
"source": [
"one_flow = X_test.iloc[[0]]\n",
"\n",
"# TODO 6: predict one_flow and convert the result to int\n",
"one_prediction = None\n",
"\n",
"print(\"Prediction:\", one_prediction)\n",
"print(\"Meaning:\", \"ATTACK\" if one_prediction == 1 else \"BENIGN\")\n"
]
},
{
"cell_type": "markdown",
"id": "a7cdf396",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "926ba101",
"metadata": {},
"outputs": [],
"source": [
"assert list(pipeline.named_steps) == [\"imputer\", \"model\"]\n",
"assert pipeline.named_steps[\"model\"].n_estimators == 100\n",
"assert len(y_pred) == len(y_test)\n",
"assert set(np.unique(y_pred)).issubset({0, 1})\n",
"assert one_prediction in {0, 1}\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "7beedd77",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. What problem does the imputer solve inside the pipeline?\n",
"2. Why is a pipeline safer than manually transforming train and test data in separate cells?\n",
"3. What does `random_state=42` contribute to the experiment?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/04_Classification_Metrics_Lab.ipynb
+236
View File
@@ -0,0 +1,236 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1ca2a12e",
"metadata": {},
"source": [
"# Module 3 — Lab 4: Evaluate an Intrusion Classifier\n",
"\n",
"\n",
"**Topics:** confusion matrix, accuracy, precision, recall, F1-score, error analysis\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "04669dcb",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- calculate a confusion matrix and classification metrics;\n",
"- distinguish false positives from false negatives;\n",
"- explain why attack recall matters in intrusion detection;\n",
"- inspect misclassified network flows.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f78f254b",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, classification_report,\n",
" confusion_matrix, precision_score, recall_score, f1_score\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "b9399f00",
"metadata": {},
"source": [
"## 1. Train the baseline pipeline\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4e0967ef",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
"df.columns = df.columns.str.strip()\n",
"df = df.replace([np.inf, -np.inf], np.nan)\n",
"\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"\n",
"pipeline = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=100, max_depth=12,\n",
" random_state=RANDOM_STATE, n_jobs=-1\n",
" )),\n",
"])\n",
"\n",
"# TODO 1: fit the pipeline and create y_pred\n",
"y_pred = None\n"
]
},
{
"cell_type": "markdown",
"id": "b83691ce",
"metadata": {},
"source": [
"## 2. Calculate the confusion matrix\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2e6d0759",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: calculate the confusion matrix and unpack tn, fp, fn, tp\n",
"cm = None\n",
"tn, fp, fn, tp = None, None, None, None\n",
"\n",
"print(\"Confusion matrix:\\n\", cm)\n",
"print(f\"TN={tn}, FP={fp}, FN={fn}, TP={tp}\")\n",
"\n",
"# TODO 3: display the confusion matrix with class names BENIGN and ATTACK\n"
]
},
{
"cell_type": "markdown",
"id": "e410267b",
"metadata": {},
"source": [
"## 3. Calculate classification metrics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bb935f4",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: calculate accuracy, attack precision, attack recall, and attack F1\n",
"accuracy = None\n",
"attack_precision = None\n",
"attack_recall = None\n",
"attack_f1 = None\n",
"\n",
"metrics = pd.Series({\n",
" \"accuracy\": accuracy,\n",
" \"attack_precision\": attack_precision,\n",
" \"attack_recall\": attack_recall,\n",
" \"attack_f1\": attack_f1,\n",
"})\n",
"print(metrics.round(3))\n",
"\n",
"# TODO 5: print classification_report with BENIGN and ATTACK target names\n"
]
},
{
"cell_type": "markdown",
"id": "c2719c46",
"metadata": {},
"source": [
"## 4. Inspect false negatives\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9b14d27c",
"metadata": {},
"outputs": [],
"source": [
"results = X_test.copy()\n",
"results[\"actual\"] = y_test\n",
"results[\"predicted\"] = y_pred\n",
"\n",
"# TODO 6: retain rows where actual is ATTACK and predicted is BENIGN\n",
"false_negatives = None\n",
"\n",
"print(\"False negatives:\", None)\n",
"false_negatives.head()\n"
]
},
{
"cell_type": "markdown",
"id": "9383f9fa",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21628fcf",
"metadata": {},
"outputs": [],
"source": [
"assert cm.shape == (2, 2)\n",
"assert tn + fp + fn + tp == len(y_test)\n",
"assert 0 <= accuracy <= 1\n",
"assert 0 <= attack_precision <= 1\n",
"assert 0 <= attack_recall <= 1\n",
"assert 0 <= attack_f1 <= 1\n",
"assert len(false_negatives) == fn\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "e2f95d64",
"metadata": {},
"source": [
"## Analyst questions\n",
"\n",
"1. Which confusion-matrix cell represents a missed attack?\n",
"2. Which metric should receive more attention if missed attacks are very costly?\n",
"3. Why can improving attack recall increase the number of false alerts?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/05_Class_Imbalance_Experiment_Lab.ipynb
+218
View File
@@ -0,0 +1,218 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "24b7696f",
"metadata": {},
"source": [
"# Module 3 — Lab 5: A Controlled Class-Imbalance Experiment\n",
"\n",
"**Topics:** class imbalance, class weights, controlled comparison, metric trade-offs\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "7ab7f5aa",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- measure the class imbalance in a security dataset;\n",
"- compare two runs of the same model on the same split;\n",
"- use `class_weight=\"balanced\"` without changing the rest of the workflow;\n",
"- select a model configuration using attack precision, recall, and F1.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "776b4b02",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix\n"
]
},
{
"cell_type": "markdown",
"id": "7386484c",
"metadata": {},
"source": [
"## 1. Prepare one fixed experiment\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61510ed9",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
"df.columns = df.columns.str.strip()\n",
"df = df.replace([np.inf, -np.inf], np.nan)\n",
"\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"\n",
"# TODO 1: calculate benign_count, attack_count, and imbalance_ratio\n",
"benign_count = None\n",
"attack_count = None\n",
"imbalance_ratio = None\n",
"\n",
"print(f\"BENIGN={benign_count}, ATTACK={attack_count}\")\n",
"print(f\"BENIGN-to-ATTACK ratio: {imbalance_ratio:.2f}:1\")\n"
]
},
{
"cell_type": "markdown",
"id": "dcec4c13",
"metadata": {},
"source": [
"## 2. Define a reusable training function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b143e2da",
"metadata": {},
"outputs": [],
"source": [
"def train_and_measure(class_weight):\n",
" # TODO 2: create the imputer + Random Forest pipeline\n",
" model = None\n",
"\n",
" # TODO 3: fit, predict, and return a metrics dictionary\n",
" return None\n"
]
},
{
"cell_type": "markdown",
"id": "e81c919c",
"metadata": {},
"source": [
"## 3. Run the controlled comparison\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5aa0bbb7",
"metadata": {},
"outputs": [],
"source": [
"# Keep the data split and all hyperparameters fixed.\n",
"# TODO 4: run the baseline with class_weight=None\n",
"baseline_metrics = None\n",
"\n",
"# TODO 5: run the weighted model with class_weight=\"balanced\"\n",
"balanced_metrics = None\n",
"\n",
"# TODO 6: create comparison_df with Baseline and Balanced rows\n",
"comparison_df = None\n",
"\n",
"comparison_df.round(3)\n"
]
},
{
"cell_type": "markdown",
"id": "7d9d295d",
"metadata": {},
"source": [
"## 4. Make an evidence-based recommendation\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e08131ac",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: select the configuration with the higher attack F1 score\n",
"recommended_configuration = None\n",
"\n",
"print(\"Recommended configuration:\", recommended_configuration)\n",
"print(comparison_df.loc[recommended_configuration].round(3))\n"
]
},
{
"cell_type": "markdown",
"id": "a6831bc3",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00a29862",
"metadata": {},
"outputs": [],
"source": [
"assert benign_count > attack_count\n",
"assert imbalance_ratio > 1\n",
"assert list(comparison_df.index) == [\"Baseline\", \"Balanced\"]\n",
"assert {\"precision\", \"recall\", \"f1\", \"false_positives\", \"false_negatives\"}.issubset(comparison_df.columns)\n",
"assert comparison_df[[\"precision\", \"recall\", \"f1\"]].apply(lambda col: col.between(0, 1).all()).all()\n",
"assert recommended_configuration in {\"Baseline\", \"Balanced\"}\n",
"print(\"✓ Lab 5 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "79d33904",
"metadata": {},
"source": [
"## Analyst questions\n",
"\n",
"1. Did class weighting improve attack recall in this run?\n",
"2. What happened to false positives and false negatives?\n",
"3. Would you choose the same configuration if analyst time were extremely limited?\n",
"4. Why is changing only one variable important in a controlled experiment?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/06_Capstone_CICIDS2017_Binary_IDS_Lab.ipynb
+289
View File
@@ -0,0 +1,289 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "291ef749",
"metadata": {},
"source": [
"# Module 3 — Lab 6: Capstone — Binary Intrusion Detection with CIC-IDS2017\n",
"\n",
"\n",
"**Topics:** end-to-end ML workflow, CIC-IDS2017, binary classification, evaluation, reporting\n",
"\n",
"> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
]
},
{
"cell_type": "markdown",
"id": "00c8ff6f",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build one complete and leakage-safe intrusion-detection workflow:\n",
"\n",
"1. load a network-flow table;\n",
"2. clean invalid values and duplicate rows;\n",
"3. map all attacks to one `ATTACK` class;\n",
"4. create one stratified train-test split;\n",
"5. train one Random Forest pipeline;\n",
"6. evaluate attack precision, recall, F1, and error counts;\n",
"7. export a short analyst report.\n",
"\n",
"The notebook uses `data/cicids2017_binary_sample.csv` when that file is available. Until the instructor prepares the official subset, it uses the included synthetic dataset and prints a clear warning.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c2228a9",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, confusion_matrix,\n",
" precision_score, recall_score, f1_score\n",
")\n",
"\n",
"REAL_DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"DEMO_DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"RANDOM_STATE = 42\n",
"\n",
"DATA_PATH = REAL_DATA_PATH if REAL_DATA_PATH.exists() else DEMO_DATA_PATH\n",
"USING_REAL_CICIDS = REAL_DATA_PATH.exists()\n",
"\n",
"print(\"Dataset:\", DATA_PATH)\n",
"if not USING_REAL_CICIDS:\n",
" print(\"WARNING: Using synthetic classroom data. Add the prepared CIC-IDS2017 file for the real capstone.\")\n"
]
},
{
"cell_type": "markdown",
"id": "c61dcfb0",
"metadata": {},
"source": [
"## 1. Load, clean, and define the binary target\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2204759",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load DATA_PATH into raw_df\n",
"raw_df = None\n",
"\n",
"# TODO 2: strip column names, remove duplicates, and replace +/- infinity with NaN\n",
"clean_df = None\n",
"\n",
"# TODO 3: create numeric X and binary y (BENIGN=0, every attack=1)\n",
"X = None\n",
"y = None\n",
"\n",
"print(f\"Raw rows: {None}\")\n",
"print(f\"Clean rows: {None}\")\n",
"print(f\"Numeric features: {None}\")\n",
"print(\"Binary class distribution:\")\n",
"print(None)\n"
]
},
{
"cell_type": "markdown",
"id": "90e14590",
"metadata": {},
"source": [
"## 2. Create one fixed train-test split\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33430abe",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: create an 80/20 split with stratify=y and random_state=42\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"print(\"Training samples:\", None)\n",
"print(\"Test samples:\", None)\n",
"print(\"Training attack rate:\", None)\n",
"print(\"Test attack rate:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "b4f6af84",
"metadata": {},
"source": [
"## 3. Build and train the baseline pipeline\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aab65468",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: create an imputer + Random Forest pipeline\n",
"baseline_pipeline = None\n",
"\n",
"# TODO 6: fit the pipeline and predict X_test\n",
"y_pred = None\n"
]
},
{
"cell_type": "markdown",
"id": "2c22c49d",
"metadata": {},
"source": [
"## 4. Evaluate the attack class\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6c6facd",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: calculate cm and unpack tn, fp, fn, tp\n",
"cm = None\n",
"tn, fp, fn, tp = None, None, None, None\n",
"\n",
"# TODO 8: calculate accuracy, attack precision, recall, and F1\n",
"baseline_metrics = None\n",
"\n",
"print(pd.Series(baseline_metrics).round(3))\n",
"print(f\"False positives: {fp} | False negatives: {fn}\")\n",
"\n",
"# TODO 9: plot and save the confusion matrix to output/capstone_confusion_matrix.png\n"
]
},
{
"cell_type": "markdown",
"id": "dc66b35c",
"metadata": {},
"source": [
"## 5. Inspect feature importance\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d1894b7",
"metadata": {},
"outputs": [],
"source": [
"# TODO 10: create a descending feature_importance Series\n",
"feature_importance = None\n",
"\n",
"print(feature_importance.head(10))\n",
"\n",
"# TODO 11: plot the top ten features and save the chart\n"
]
},
{
"cell_type": "markdown",
"id": "9f05106e",
"metadata": {},
"source": [
"## 6. Export metrics and write an analyst summary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e66b54f",
"metadata": {},
"outputs": [],
"source": [
"# TODO 12: export baseline_metrics to output/capstone_metrics.csv\n",
"\n",
"# TODO 13: create a summary with dataset type, row count, feature count,\n",
"# attack rate, precision, recall, F1, false positives, false negatives,\n",
"# and a recommendation. Save it to output/capstone_summary.txt.\n",
"summary = None\n",
"\n",
"print(summary)\n"
]
},
{
"cell_type": "markdown",
"id": "9b0c0e2f",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b28ca2a2",
"metadata": {},
"outputs": [],
"source": [
"assert isinstance(clean_df, pd.DataFrame)\n",
"assert set(y.unique()) == {0, 1}\n",
"assert X.shape[1] >= 5\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert cm.shape == (2, 2)\n",
"assert len(feature_importance) == X.shape[1]\n",
"assert (OUTPUT_DIR / \"capstone_confusion_matrix.png\").exists()\n",
"assert (OUTPUT_DIR / \"capstone_feature_importance.png\").exists()\n",
"assert (OUTPUT_DIR / \"capstone_metrics.csv\").exists()\n",
"assert (OUTPUT_DIR / \"capstone_summary.txt\").exists()\n",
"print(\"✓ Capstone lab completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "c7bf86d2",
"metadata": {},
"source": [
"## Capstone questions\n",
"\n",
"1. How many attacks did the model miss, and why does that number matter operationally?\n",
"2. Which features received the highest importance? Do they make sense for network flows?\n",
"3. Why does random row splitting provide an optimistic estimate for some security datasets?\n",
"4. What additional validation would you require before deploying this detector?\n",
"5. If the notebook used synthetic data, which conclusions must be rechecked on the official CIC-IDS2017 subset?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Practical/data/cicids2017_binary_sample.csv
+180001
View File
File diff suppressed because it is too large Load Diff
Lab-3/Lab-3-Practical/data/synthetic_security_flows.csv
+1205
View File
File diff suppressed because it is too large Load Diff
Lab-3/Lab-3-Solution.zip
BIN
View File
Binary file not shown.
Lab-3/Lab-3-Solution/01_Security_Features_Labels_Solution.ipynb
+227
View File
@@ -0,0 +1,227 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ce4d0249",
"metadata": {},
"source": [
"# Module 3 — Lab 1: Security Features and Binary Labels\n",
"\n",
" \n",
"**Topics:** samples, features, labels, class distribution\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "c70b2987",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- identify samples, features, and labels in a security dataset;\n",
"- inspect the schema and data types of a network-flow table;\n",
"- convert multiple attack names into one binary target;\n",
"- explain why class distribution matters before model training.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dd82f5a2",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
"RANDOM_STATE = 42\n"
]
},
{
"cell_type": "markdown",
"id": "6345844b",
"metadata": {},
"source": [
"## 1. Load and inspect the flow table\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b8d885cb",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH)\n",
"\n",
"print(\"Shape:\", df.shape)\n",
"df.head()\n"
]
},
{
"cell_type": "markdown",
"id": "19535e33",
"metadata": {},
"source": [
"## 2. Identify the label and numeric features\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f87f05f8",
"metadata": {},
"outputs": [],
"source": [
"y_text = df[\"Label\"].astype(str).str.strip()\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"\n",
"print(\"Feature matrix shape:\", X.shape)\n",
"print(\"Original labels:\")\n",
"print(y_text.value_counts())\n"
]
},
{
"cell_type": "markdown",
"id": "59f3f674",
"metadata": {},
"source": [
"## 3. Create a binary security target\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b4d0c24e",
"metadata": {},
"outputs": [],
"source": [
"normalized_labels = y_text.str.strip().str.upper()\n",
"y_binary = normalized_labels.ne(\"BENIGN\").astype(int)\n",
"class_counts = y_binary.value_counts().sort_index()\n",
"\n",
"print(class_counts.rename(index={0: \"BENIGN\", 1: \"ATTACK\"}))\n",
"print(\"Attack rate:\", round(y_binary.mean(), 3))\n"
]
},
{
"cell_type": "markdown",
"id": "28d40643",
"metadata": {},
"source": [
"## 4. Connect rows to feature vectors\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0506346",
"metadata": {},
"outputs": [],
"source": [
"sample_index = 0\n",
"sample_features = X.iloc[sample_index]\n",
"sample_target = int(y_binary.iloc[sample_index])\n",
"\n",
"print(\"Selected feature vector:\")\n",
"print(sample_features)\n",
"print(\"Binary target:\", sample_target)\n"
]
},
{
"cell_type": "markdown",
"id": "3d4855d6",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4159047f",
"metadata": {},
"outputs": [],
"source": [
"assert isinstance(df, pd.DataFrame)\n",
"assert \"Label\" not in X.columns\n",
"assert set(y_binary.unique()) == {0, 1}\n",
"assert len(X) == len(y_binary)\n",
"assert class_counts.loc[0] > class_counts.loc[1]\n",
"assert len(sample_features) == X.shape[1]\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "a66bcc4e",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is each row considered one sample while each column is considered one feature?\n",
"2. What information is lost when several attack families are mapped to one `ATTACK` class?\n",
"3. Why could high accuracy be misleading when `BENIGN` is the majority class?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5b4575cc-2166-474b-bd0e-b49d94d1cfc2",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d03b205-5584-41c8-baaf-272b32bdf0b2",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ecfa6b2-cb34-45cc-98b4-5417f48f9f82",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "40ddd666-9f13-46bd-8a58-45bef952bfa2",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Solution/02_Clean_Split_No_Leakage_Solution.ipynb
+203
View File
@@ -0,0 +1,203 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "8aacddc0",
"metadata": {},
"source": [
"# Module 3 — Lab 2: Clean, Split, and Prevent Data Leakage\n",
"\n",
"\n",
"**Topics:** duplicates, missing values, train-test split, imputation, standardization, leakage\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "f26e6221",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- remove duplicate records and normalize invalid numeric values;\n",
"- create a stratified train-test split;\n",
"- fit imputation and scaling operations on training data only;\n",
"- explain how fitting preprocessing on the test set causes leakage.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e440f789",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.preprocessing import StandardScaler\n"
]
},
{
"cell_type": "markdown",
"id": "28421713",
"metadata": {},
"source": [
"## 1. Load and clean the raw table\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "797874c4",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(DATA_PATH)\n",
"raw_df.columns = raw_df.columns.str.strip()\n",
"clean_df = raw_df.drop_duplicates().copy()\n",
"clean_df = clean_df.replace([np.inf, -np.inf], np.nan)\n",
"\n",
"print(\"Raw rows:\", len(raw_df))\n",
"print(\"Clean rows:\", len(clean_df))\n",
"print(\"Missing numeric values:\", int(clean_df.select_dtypes(include=\"number\").isna().sum().sum()))\n"
]
},
{
"cell_type": "markdown",
"id": "17f8834a",
"metadata": {},
"source": [
"## 2. Create features, target, and a stratified split\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b0b8cf4",
"metadata": {},
"outputs": [],
"source": [
"X = clean_df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = clean_df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.2,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"\n",
"print(\"Train shape:\", X_train.shape)\n",
"print(\"Test shape:\", X_test.shape)\n",
"print(\"Train attack rate:\", round(y_train.mean(), 3))\n",
"print(\"Test attack rate:\", round(y_test.mean(), 3))\n"
]
},
{
"cell_type": "markdown",
"id": "a742c62a",
"metadata": {},
"source": [
"## 3. Fit preprocessing on training data only\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43a2c168",
"metadata": {},
"outputs": [],
"source": [
"imputer = SimpleImputer(strategy=\"median\")\n",
"scaler = StandardScaler()\n",
"\n",
"X_train_imputed = imputer.fit_transform(X_train)\n",
"X_test_imputed = imputer.transform(X_test)\n",
"\n",
"X_train_scaled = scaler.fit_transform(X_train_imputed)\n",
"X_test_scaled = scaler.transform(X_test_imputed)\n",
"\n",
"print(\"Mean of scaled training columns:\", np.round(X_train_scaled.mean(axis=0), 3))\n",
"print(\"Mean of scaled test columns:\", np.round(X_test_scaled.mean(axis=0), 3))\n"
]
},
{
"cell_type": "markdown",
"id": "0c3fc6ca",
"metadata": {},
"source": [
"## Why the test mean is not exactly zero\n",
"\n",
"`StandardScaler` learned its mean and standard deviation from the training set. The test set is transformed with those same values. A non-zero test mean is expected and is evidence that the test set was not used to fit the scaler.\n"
]
},
{
"cell_type": "markdown",
"id": "459304bb",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "478a41eb",
"metadata": {},
"outputs": [],
"source": [
"assert len(clean_df) <= len(raw_df)\n",
"assert np.isfinite(X_train_scaled).all()\n",
"assert np.isfinite(X_test_scaled).all()\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert np.allclose(X_train_scaled.mean(axis=0), 0.0, atol=1e-10)\n",
"assert X_train_scaled.shape[1] == X.shape[1]\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "ca45be08",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must the imputer learn medians from the training set only?\n",
"2. What information would leak if the scaler were fit before the train-test split?\n",
"3. Why is stratification useful for an imbalanced security target?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Solution/03_Random_Forest_Pipeline_Solution.ipynb
+252
View File
@@ -0,0 +1,252 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1f325e4e",
"metadata": {},
"source": [
"# Module 3 — Lab 3: A Random Forest Security Pipeline\n",
"\n",
"**Topics:** Scikit-learn Pipeline, imputation, Random Forest, fit, predict\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "e9ba9f17",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- create one reusable preprocessing-and-model pipeline;\n",
"- train a Random Forest binary classifier;\n",
"- generate predictions for unseen data;\n",
"- explain why tree models do not require feature standardization.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dc04708",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n"
]
},
{
"cell_type": "markdown",
"id": "3746fff2",
"metadata": {},
"source": [
"## 1. Prepare the binary classification data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "96d91cc2",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH)\n",
"df.columns = df.columns.str.strip()\n",
"df = df.drop_duplicates().replace([np.inf, -np.inf], np.nan)\n",
"\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.2,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "69eb8c19",
"metadata": {},
"source": [
"## 2. Build one pipeline\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0dbae88b",
"metadata": {},
"outputs": [],
"source": [
"pipeline = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=100,\n",
" max_depth=12,\n",
" random_state=RANDOM_STATE,\n",
" n_jobs=-1,\n",
" )),\n",
"])\n",
"\n",
"print(pipeline)\n"
]
},
{
"cell_type": "markdown",
"id": "ea59d402",
"metadata": {},
"source": [
"## Why there is no `StandardScaler`\n",
"\n",
"Random Forest makes decisions by comparing feature values with thresholds. Scaling a column does not change the ordering of its values, so standardization is not required for this model family. The imputer remains inside the pipeline because the model needs finite values.\n"
]
},
{
"cell_type": "markdown",
"id": "c8f9a97a",
"metadata": {},
"source": [
"## 3. Train and predict\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c6194b17",
"metadata": {},
"outputs": [],
"source": [
"pipeline.fit(X_train, y_train)\n",
"y_pred = pipeline.predict(X_test)\n",
"\n",
"print(\"Predictions:\", y_pred[:20])\n",
"print(\"Number of test samples:\", len(y_pred))\n"
]
},
{
"cell_type": "markdown",
"id": "b5ad6c87",
"metadata": {},
"source": [
"## 4. Predict one flow\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7eca199f",
"metadata": {},
"outputs": [],
"source": [
"one_flow = X_test.iloc[[0]]\n",
"one_prediction = int(pipeline.predict(one_flow)[0])\n",
"\n",
"print(\"Prediction:\", one_prediction)\n",
"print(\"Meaning:\", \"ATTACK\" if one_prediction == 1 else \"BENIGN\")\n"
]
},
{
"cell_type": "markdown",
"id": "90d72108",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "98f48da9",
"metadata": {},
"outputs": [],
"source": [
"assert list(pipeline.named_steps) == [\"imputer\", \"model\"]\n",
"assert pipeline.named_steps[\"model\"].n_estimators == 100\n",
"assert len(y_pred) == len(y_test)\n",
"assert set(np.unique(y_pred)).issubset({0, 1})\n",
"assert one_prediction in {0, 1}\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "6100563d",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. What problem does the imputer solve inside the pipeline?\n",
"2. Why is a pipeline safer than manually transforming train and test data in separate cells?\n",
"3. What does `random_state=42` contribute to the experiment?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aeab0fb6-54bf-4e88-87bc-400d658d67b7",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "75b25828-0978-465b-a789-3181b0c25f38",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "8b198739-c8b0-4355-a44e-08f9c205768b",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "10091acc-3fbd-4e91-9bb4-f994516ce995",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Solution/04_Classification_Metrics_Solution.ipynb
+245
View File
@@ -0,0 +1,245 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "76db28c7",
"metadata": {},
"source": [
"# Module 3 — Lab 4: Evaluate an Intrusion Classifier\n",
"\n",
"\n",
"**Topics:** confusion matrix, accuracy, precision, recall, F1-score, error analysis\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "778202bf",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- calculate a confusion matrix and classification metrics;\n",
"- distinguish false positives from false negatives;\n",
"- explain why attack recall matters in intrusion detection;\n",
"- inspect misclassified network flows.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "91e09f7c",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/cicids2017_binary_sample.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, classification_report,\n",
" confusion_matrix, precision_score, recall_score, f1_score\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "29e0b207",
"metadata": {},
"source": [
"## 1. Train the baseline pipeline\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f53d7241",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
"df.columns = df.columns.str.strip()\n",
"df = df.replace([np.inf, -np.inf], np.nan)\n",
"\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"\n",
"pipeline = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=100, max_depth=12,\n",
" random_state=RANDOM_STATE, n_jobs=-1\n",
" )),\n",
"])\n",
"\n",
"pipeline.fit(X_train, y_train)\n",
"y_pred = pipeline.predict(X_test)\n"
]
},
{
"cell_type": "markdown",
"id": "0781cb26",
"metadata": {},
"source": [
"## 2. Calculate the confusion matrix\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21075b2f",
"metadata": {},
"outputs": [],
"source": [
"cm = confusion_matrix(y_test, y_pred)\n",
"tn, fp, fn, tp = cm.ravel()\n",
"\n",
"print(\"Confusion matrix:\\n\", cm)\n",
"print(f\"TN={tn}, FP={fp}, FN={fn}, TP={tp}\")\n",
"\n",
"ConfusionMatrixDisplay(\n",
" confusion_matrix=cm,\n",
" display_labels=[\"BENIGN\", \"ATTACK\"],\n",
").plot(cmap=\"Blues\")\n",
"plt.title(\"Baseline intrusion classifier\")\n",
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"id": "223681c3",
"metadata": {},
"source": [
"## 3. Calculate classification metrics\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a5a1597a",
"metadata": {},
"outputs": [],
"source": [
"accuracy = accuracy_score(y_test, y_pred)\n",
"attack_precision = precision_score(y_test, y_pred, zero_division=0)\n",
"attack_recall = recall_score(y_test, y_pred, zero_division=0)\n",
"attack_f1 = f1_score(y_test, y_pred, zero_division=0)\n",
"\n",
"metrics = pd.Series({\n",
" \"accuracy\": accuracy,\n",
" \"attack_precision\": attack_precision,\n",
" \"attack_recall\": attack_recall,\n",
" \"attack_f1\": attack_f1,\n",
"})\n",
"print(metrics.round(3))\n",
"\n",
"print(classification_report(\n",
" y_test,\n",
" y_pred,\n",
" target_names=[\"BENIGN\", \"ATTACK\"],\n",
" zero_division=0,\n",
"))\n"
]
},
{
"cell_type": "markdown",
"id": "ebbfff71",
"metadata": {},
"source": [
"## 4. Inspect false negatives\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4adf798b",
"metadata": {},
"outputs": [],
"source": [
"results = X_test.copy()\n",
"results[\"actual\"] = y_test\n",
"results[\"predicted\"] = y_pred\n",
"\n",
"false_negatives = results[\n",
" (results[\"actual\"] == 1) & (results[\"predicted\"] == 0)\n",
"]\n",
"\n",
"print(\"False negatives:\", len(false_negatives))\n",
"false_negatives.head()\n"
]
},
{
"cell_type": "markdown",
"id": "4cba4276",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0a64652b",
"metadata": {},
"outputs": [],
"source": [
"assert cm.shape == (2, 2)\n",
"assert tn + fp + fn + tp == len(y_test)\n",
"assert 0 <= accuracy <= 1\n",
"assert 0 <= attack_precision <= 1\n",
"assert 0 <= attack_recall <= 1\n",
"assert 0 <= attack_f1 <= 1\n",
"assert len(false_negatives) == fn\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "8f5b93d9",
"metadata": {},
"source": [
"## Analyst questions\n",
"\n",
"1. Which confusion-matrix cell represents a missed attack?\n",
"2. Which metric should receive more attention if missed attacks are very costly?\n",
"3. Why can improving attack recall increase the number of false alerts?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Solution/05_Class_Imbalance_Experiment_Solution.ipynb
+230
View File
@@ -0,0 +1,230 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a3a90183",
"metadata": {},
"source": [
"# Module 3 — Lab 5: A Controlled Class-Imbalance Experiment\n",
"\n",
" \n",
"**Topics:** class imbalance, class weights, controlled comparison, metric trade-offs\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "4d80ad41",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- measure the class imbalance in a security dataset;\n",
"- compare two runs of the same model on the same split;\n",
"- use `class_weight=\"balanced\"` without changing the rest of the workflow;\n",
"- select a model configuration using attack precision, recall, and F1.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61a4b369",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/synthetic_security_flows.csv\")\n",
"RANDOM_STATE = 42\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix\n"
]
},
{
"cell_type": "markdown",
"id": "1301b149",
"metadata": {},
"source": [
"## 1. Prepare one fixed experiment\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f5ea362d",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH).drop_duplicates()\n",
"df.columns = df.columns.str.strip()\n",
"df = df.replace([np.inf, -np.inf], np.nan)\n",
"\n",
"X = df.drop(columns=\"Label\").select_dtypes(include=\"number\")\n",
"y = df[\"Label\"].astype(str).str.strip().str.upper().ne(\"BENIGN\").astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"\n",
"benign_count = int((y_train == 0).sum())\n",
"attack_count = int((y_train == 1).sum())\n",
"imbalance_ratio = benign_count / attack_count\n",
"\n",
"print(f\"BENIGN={benign_count}, ATTACK={attack_count}\")\n",
"print(f\"BENIGN-to-ATTACK ratio: {imbalance_ratio:.2f}:1\")\n"
]
},
{
"cell_type": "markdown",
"id": "e3dfcb43",
"metadata": {},
"source": [
"## 2. Define a reusable training function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51c588dd",
"metadata": {},
"outputs": [],
"source": [
"def train_and_measure(class_weight):\n",
" model = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=100,\n",
" max_depth=12,\n",
" class_weight=class_weight,\n",
" random_state=RANDOM_STATE,\n",
" n_jobs=-1,\n",
" )),\n",
" ])\n",
" model.fit(X_train, y_train)\n",
" predictions = model.predict(X_test)\n",
" tn, fp, fn, tp = confusion_matrix(y_test, predictions).ravel()\n",
" return {\n",
" \"precision\": precision_score(y_test, predictions, zero_division=0),\n",
" \"recall\": recall_score(y_test, predictions, zero_division=0),\n",
" \"f1\": f1_score(y_test, predictions, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" }\n"
]
},
{
"cell_type": "markdown",
"id": "c81e9645",
"metadata": {},
"source": [
"## 3. Run the controlled comparison\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac3530d7",
"metadata": {},
"outputs": [],
"source": [
"baseline_metrics = train_and_measure(class_weight=None)\n",
"balanced_metrics = train_and_measure(class_weight=\"balanced\")\n",
"\n",
"comparison_df = pd.DataFrame(\n",
" [baseline_metrics, balanced_metrics],\n",
" index=[\"Baseline\", \"Balanced\"],\n",
")\n",
"\n",
"comparison_df.round(3)\n"
]
},
{
"cell_type": "markdown",
"id": "e4f99ffd",
"metadata": {},
"source": [
"## 4. Make an evidence-based recommendation\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc7c44fb",
"metadata": {},
"outputs": [],
"source": [
"recommended_configuration = comparison_df[\"f1\"].idxmax()\n",
"\n",
"print(\"Recommended configuration:\", recommended_configuration)\n",
"print(comparison_df.loc[recommended_configuration].round(3))\n"
]
},
{
"cell_type": "markdown",
"id": "c6e2a54c",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c6389320",
"metadata": {},
"outputs": [],
"source": [
"assert benign_count > attack_count\n",
"assert imbalance_ratio > 1\n",
"assert list(comparison_df.index) == [\"Baseline\", \"Balanced\"]\n",
"assert {\"precision\", \"recall\", \"f1\", \"false_positives\", \"false_negatives\"}.issubset(comparison_df.columns)\n",
"assert comparison_df[[\"precision\", \"recall\", \"f1\"]].apply(lambda col: col.between(0, 1).all()).all()\n",
"assert recommended_configuration in {\"Baseline\", \"Balanced\"}\n",
"print(\"✓ Lab 5 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "69c051ee",
"metadata": {},
"source": [
"## Analyst questions\n",
"\n",
"1. Did class weighting improve attack recall in this run?\n",
"2. What happened to false positives and false negatives?\n",
"3. Would you choose the same configuration if analyst time were extremely limited?\n",
"4. Why is changing only one variable important in a controlled experiment?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-3/Lab-3-Solution/06_Capstone_CICIDS2017_Binary_IDS_Solution.ipynb
+461
View File
File diff suppressed because one or more lines are too long
Lab-3/Lab-3-Solution/data/cicids2017_binary_sample.csv
+180001
View File
File diff suppressed because it is too large Load Diff
Lab-3/Lab-3-Solution/data/synthetic_security_flows.csv
+1205
View File
File diff suppressed because it is too large Load Diff
Lab-4/Lab-4-Practical.zip
BIN
View File
Binary file not shown.
Lab-4/Lab-4-Practical/01_SMS_Dataset_Inspection_Lab.ipynb
+235
View File
@@ -0,0 +1,235 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1545411a",
"metadata": {},
"source": [
"# Module 4 — Lab 1: Inspecting the SMS Spam Dataset\n",
"\n",
" \n",
"**Topics:** text data, duplicates, conflicting labels, class distribution\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "ac8bdc2a",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load the original UCI tab-separated text file;\n",
"- audit missing values, exact duplicates, and conflicting labels;\n",
"- create a binary target without modifying the message text;\n",
"- describe class imbalance and message-length differences;\n",
"- explain why aggressive cleaning may remove useful security signals.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c903f90d",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download \"\n",
" \"and extract the official UCI SMS Spam Collection.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "0dc48c80",
"metadata": {},
"source": [
"## 1. Load the original collection\n",
"\n",
"The source file has no header. Each row contains a label, one tab character, and the message text.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bcba4fd6",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the tab-separated UCI file into raw_df.\n",
"# Use the column names label and message.\n",
"raw_df = None\n",
"\n",
"# TODO 2: display the shape, first five rows, and raw label counts.\n",
"print(\"Raw shape:\", None)\n",
"raw_df\n"
]
},
{
"cell_type": "markdown",
"id": "585072d3",
"metadata": {},
"source": [
"## 2. Audit the records\n",
"\n",
"Exact duplicates can cross a random train-test split and make evaluation too optimistic. A more serious problem occurs when identical text has different labels. Audit both conditions before modeling.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "72d9609c",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: normalize label whitespace and remove rows with missing or empty text.\n",
"audited_df = None\n",
"\n",
"# TODO 4: count exact duplicate rows.\n",
"exact_duplicate_count = None\n",
"\n",
"# TODO 5: find messages associated with more than one distinct label.\n",
"conflicting_messages = None\n",
"\n",
"print(\"Exact duplicate rows:\", exact_duplicate_count)\n",
"print(\"Conflicting message texts:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "06e3d7d2",
"metadata": {},
"source": [
"## 3. Build the clean classroom table\n",
"\n",
"Remove ambiguous messages before exact duplicates, then map `ham → 0` and `spam → 1`. Keep the original text intact.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3bf6d47d",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: remove every conflicting message and then remove exact duplicates.\n",
"clean_df = None\n",
"\n",
"# TODO 7: map ham to 0 and spam to 1 in a new target column.\n",
"clean_df[\"target\"] = None\n",
"\n",
"print(\"Clean shape:\", None)\n",
"print(clean_df[\"target\"].value_counts().sort_index())\n"
]
},
{
"cell_type": "markdown",
"id": "abd58b67",
"metadata": {},
"source": [
"## 4. Describe the classes and message lengths\n",
"\n",
"Length is used only for exploration. The later models will learn from the message text itself.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ae01028",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: create character_count and word_count columns.\n",
"clean_df[\"character_count\"] = None\n",
"clean_df[\"word_count\"] = None\n",
"\n",
"# TODO 9: calculate a summary indexed by label with count and median lengths.\n",
"length_summary = None\n",
"class_distribution = None\n",
"\n",
"print(length_summary)\n",
"print(\"Class proportions:\")\n",
"print(class_distribution)\n"
]
},
{
"cell_type": "markdown",
"id": "ca6f77a2",
"metadata": {},
"source": [
"## Security note: do not over-clean\n",
"\n",
"URLs, phone numbers, currency symbols, punctuation, unusual capitalization, and deliberate misspellings may contain spam evidence. Begin with the vectorizer defaults. Add normalization only when an experiment demonstrates that it improves validation performance without hiding operationally important signals.\n"
]
},
{
"cell_type": "markdown",
"id": "9559352e",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb0dabd7",
"metadata": {},
"outputs": [],
"source": [
"assert list(raw_df.columns) == [\"label\", \"message\"]\n",
"assert clean_df[\"message\"].notna().all()\n",
"assert clean_df[\"message\"].ne(\"\").all()\n",
"assert set(clean_df[\"label\"].unique()) == {\"ham\", \"spam\"}\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert not clean_df[\"message\"].isin(conflicting_messages).any()\n",
"assert class_distribution.loc[\"ham\"] > class_distribution.loc[\"spam\"]\n",
"assert {\"character_count\", \"word_count\"}.issubset(clean_df.columns)\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "62472b1f",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why should exact duplicates be removed before the train-test split?\n",
"2. Why might removing URLs and currency symbols reduce detection quality?\n",
"3. Why is accuracy alone risky for this dataset?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/02_Bag_of_Words_TFIDF_Lab.ipynb
+263
View File
@@ -0,0 +1,263 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ba1236aa",
"metadata": {},
"source": [
"# Module 4 — Lab 2: Bag of Words and TF-IDF\n",
"\n",
" \n",
"**Topics:** CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "c66017ea",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- explain how Bag of Words maps documents to numeric vectors;\n",
"- inspect a document-term matrix and its vocabulary;\n",
"- fit TF-IDF on training text and transform unseen text;\n",
"- describe why text matrices are high-dimensional and sparse;\n",
"- compare unigram features with unigram-plus-bigram features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4a39b50",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "e5850d02",
"metadata": {},
"source": [
"## 1. Make Bag of Words visible\n",
"\n",
"Use three short messages so that every row, token, and count can be inspected directly.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c21fbff3",
"metadata": {},
"outputs": [],
"source": [
"tiny_messages = [\n",
" \"free prize now\",\n",
" \"free meeting today\",\n",
" \"project meeting today\",\n",
"]\n",
"\n",
"# TODO 1: fit a CountVectorizer on tiny_messages and transform the messages.\n",
"count_vectorizer = None\n",
"tiny_bow = None\n",
"\n",
"# TODO 2: create a DataFrame whose columns are the learned feature names.\n",
"tiny_bow_df = None\n",
"tiny_bow_df\n"
]
},
{
"cell_type": "markdown",
"id": "92098ba9",
"metadata": {},
"source": [
"## 2. Prepare the UCI messages\n",
"\n",
"Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d0fc369e",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"\n",
"conflicts = df.groupby(\"message\")[\"label\"].nunique()\n",
"conflict_texts = conflicts[conflicts.gt(1)].index\n",
"df = df[~df[\"message\"].isin(conflict_texts)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"print(\"Clean messages:\", len(df))\n"
]
},
{
"cell_type": "markdown",
"id": "c01ce3be",
"metadata": {},
"source": [
"## 3. Fit TF-IDF without leakage\n",
"\n",
"`fit_transform` learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only `transform`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "12627cb9",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: split message and target into 80% training and 20% test data.\n",
"# Use random_state=42 and stratify=y.\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"# TODO 4: fit TfidfVectorizer on X_train only and transform X_test.\n",
"tfidf_vectorizer = None\n",
"X_train_tfidf = None\n",
"X_test_tfidf = None\n",
"\n",
"print(\"Training matrix shape:\", None)\n",
"print(\"Test matrix shape:\", None)\n",
"print(\"Training density:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "7c6ee2c2",
"metadata": {},
"source": [
"## 4. Inspect the learned feature space\n",
"\n",
"A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "42745259",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: calculate each term's mean TF-IDF weight across training messages.\n",
"mean_tfidf = None\n",
"\n",
"# TODO 6: build a descending Series and keep the 15 highest-weight terms.\n",
"top_terms = None\n",
"top_terms\n"
]
},
{
"cell_type": "markdown",
"id": "d60a97ed",
"metadata": {},
"source": [
"## 5. Add short phrases with bigrams\n",
"\n",
"A unigram represents one token. A bigram can preserve short phrases such as `call now` or `free entry`, but it also increases the number of features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b6458f8",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: fit a second TF-IDF vectorizer with unigrams and bigrams.\n",
"ngram_vectorizer = None\n",
"X_train_ngram = None\n",
"\n",
"print(\"Unigram features:\", None)\n",
"print(\"Unigram + bigram features:\", None)\n",
"print(\"Example bigrams:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "5703a0a9",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "46da5f31",
"metadata": {},
"outputs": [],
"source": [
"assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))\n",
"assert tiny_bow_df.loc[\"message_1\", \"free\"] == 1\n",
"assert len(X_train) + len(X_test) == len(df)\n",
"assert X_train_tfidf.shape[0] == len(X_train)\n",
"assert X_test_tfidf.shape[0] == len(X_test)\n",
"assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]\n",
"assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]\n",
"assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]\n",
"assert len(top_terms) == 15\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "31bb9a13",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is a text feature matrix usually sparse?\n",
"2. Why must the vectorizer be fitted only on training messages?\n",
"3. What trade-off is introduced by adding bigrams?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/03_Naive_Bayes_Baseline_Lab.ipynb
+274
View File
@@ -0,0 +1,274 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ac0bd1d0",
"metadata": {},
"source": [
"# Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline\n",
"\n",
"\n",
"**Topics:** Pipeline, MultinomialNB, confusion matrix, baseline, error analysis\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "e0a933d7",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- place TF-IDF and Multinomial Naive Bayes in one pipeline;\n",
"- explain how the pipeline prevents preprocessing leakage;\n",
"- calculate spam precision, recall, and F1;\n",
"- compare a trained classifier with a naive baseline;\n",
"- inspect false positives and false negatives.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43e5d008",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "831471be",
"metadata": {},
"source": [
"## 1. Load, audit, and split the messages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32983efc",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"print(\"Training messages:\", len(X_train))\n",
"print(\"Test messages:\", len(X_test))\n",
"print(\"Training spam rate:\", f\"{y_train.mean():.2%}\")\n",
"print(\"Test spam rate:\", f\"{y_test.mean():.2%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "8bd01144",
"metadata": {},
"source": [
"## 2. Build and train one pipeline\n",
"\n",
"The pipeline fits TF-IDF only on `X_train`. During prediction, it applies the already-fitted transformation to `X_test`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "458119d7",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: create a Pipeline named nb_pipeline with a TF-IDF vectorizer\n",
"# followed by MultinomialNB(alpha=1.0).\n",
"nb_pipeline = None\n",
"\n",
"# TODO 2: fit the pipeline on X_train and y_train, then predict X_test.\n",
"y_pred = None\n",
"\n",
"print(nb_pipeline)\n"
]
},
{
"cell_type": "markdown",
"id": "8aedf1e8",
"metadata": {},
"source": [
"## 3. Evaluate the spam class\n",
"\n",
"- **Spam precision:** when the model blocks a message, how often is it truly spam?\n",
"- **Spam recall:** of all spam messages, how many did the model catch?\n",
"- **Spam F1:** harmonic mean of spam precision and recall.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "315b9a3e",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: calculate the confusion matrix and unpack tn, fp, fn, tp.\n",
"cm = None\n",
"tn, fp, fn, tp = None, None, None, None\n",
"\n",
"# TODO 4: calculate accuracy and spam precision, recall, and F1.\n",
"nb_metrics = None\n",
"\n",
"print(pd.Series(nb_metrics).round(4))\n",
"print(\"\\nClassification report:\")\n",
"print(None)\n"
]
},
{
"cell_type": "markdown",
"id": "0238d75f",
"metadata": {},
"source": [
"## 4. Compare with a naive baseline\n",
"\n",
"Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac5b05ca",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: train a most_frequent DummyClassifier and predict X_test.\n",
"dummy = None\n",
"dummy_pred = None\n",
"\n",
"# TODO 6: compare the two models using spam F1 and accuracy.\n",
"comparison = None\n",
"comparison\n"
]
},
{
"cell_type": "markdown",
"id": "198f66ac",
"metadata": {},
"source": [
"## 5. Inspect operational errors\n",
"\n",
"A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ae6e63f",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: create a DataFrame with message, actual, and predicted columns.\n",
"error_analysis = None\n",
"\n",
"# TODO 8: keep only incorrect predictions and add an error_type column\n",
"# containing false_positive or false_negative.\n",
"errors = None\n",
"\n",
"print(errors[\"error_type\"].value_counts())\n",
"errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "1b3715ec",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1962e1aa",
"metadata": {},
"outputs": [],
"source": [
"assert list(nb_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert isinstance(nb_pipeline.named_steps[\"model\"], MultinomialNB)\n",
"assert len(y_pred) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert set(nb_metrics) == {\"accuracy\", \"spam_precision\", \"spam_recall\", \"spam_f1\"}\n",
"assert comparison.loc[\"Most-frequent baseline\", \"spam_f1\"] == 0\n",
"assert set(errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "2d1ea84b",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is the most-frequent classifier an important baseline?\n",
"2. Which operational harm is represented by a false positive?\n",
"3. Why should TF-IDF and the model be placed in one pipeline?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/04_Linear_Model_Comparison_Lab.ipynb
+279
View File
@@ -0,0 +1,279 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4c3ac6bc",
"metadata": {},
"source": [
"# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
"\n",
"\n",
"**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "fb25bc5b",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare several classifiers on one fixed train-test split;\n",
"- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
"- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
"- connect false positives and false negatives to operational consequences;\n",
"- explain why kernel SVM is not required for this classroom workflow.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "603330c3",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression, Perceptron\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.svm import LinearSVC\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "cae34196",
"metadata": {},
"source": [
"## 1. Prepare one fixed experiment\n",
"\n",
"Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "1f43dbfe",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "2e23c9db",
"metadata": {},
"source": [
"## 2. Define the candidate pipelines\n",
"\n",
"All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5af7ba3e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Models: None\n"
]
}
],
"source": [
"# TODO 1: create four pipelines with identical TF-IDF settings.\n",
"# Models: MultinomialNB, Perceptron, LinearSVC, and LogisticRegression.\n",
"models = None\n",
"\n",
"print(\"Models:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "86aa5371",
"metadata": {},
"source": [
"## 3. Train and evaluate fairly\n",
"\n",
"Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "dd93163a",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: fit every pipeline on the same training split.\n",
"# TODO 3: record accuracy, spam precision, spam recall, spam F1,\n",
"# false positives, and false negatives for each model.\n",
"fitted_models = None\n",
"predictions = None\n",
"comparison = None\n",
"\n",
"comparison\n"
]
},
{
"cell_type": "markdown",
"id": "1b281735",
"metadata": {},
"source": [
"## 4. Select and inspect the strongest model\n",
"\n",
"For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "b327c2c6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best model by spam F1: None\n",
"Number of errors: None\n"
]
},
{
"ename": "AttributeError",
"evalue": "'NoneType' object has no attribute 'head'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[5], line 11\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mBest model by spam F1:\u001b[39m\u001b[38;5;124m\"\u001b[39m, best_model_name)\n\u001b[1;32m 10\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNumber of errors:\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m---> 11\u001b[0m \u001b[43mbest_errors\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mhead\u001b[49m(\u001b[38;5;241m10\u001b[39m)\n",
"\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'head'"
]
}
],
"source": [
"# TODO 4: select the model with the highest spam F1.\n",
"best_model_name = None\n",
"best_pipeline = None\n",
"best_predictions = None\n",
"\n",
"# TODO 5: create a DataFrame containing the best model's test errors.\n",
"best_errors = None\n",
"\n",
"print(\"Best model by spam F1:\", best_model_name)\n",
"print(\"Number of errors:\", None)\n",
"best_errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "e2eb368a",
"metadata": {},
"source": [
"## Interpretation checkpoint\n",
"\n",
"A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
]
},
{
"cell_type": "markdown",
"id": "e39f0a30",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4666cc68",
"metadata": {},
"outputs": [],
"source": [
"assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
"assert set(fitted_models) == set(models)\n",
"assert set(predictions) == set(models)\n",
"assert comparison.shape == (4, 6)\n",
"assert comparison.index.is_unique\n",
"assert comparison[\"spam_f1\"].between(0, 1).all()\n",
"assert best_model_name in models\n",
"assert len(best_predictions) == len(y_test)\n",
"assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "9e7f451c",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must every model use the same data split?\n",
"2. Why is a linear SVM appropriate for TF-IDF features?\n",
"3. Why might the highest-accuracy model still be unsuitable for spam filtering?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/05_Probability_Thresholds_Interpretation_Lab.ipynb
+305
View File
@@ -0,0 +1,305 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "64dbadae",
"metadata": {},
"source": [
"# Module 4 — Lab 5: Probability, Thresholds, and Interpretation\n",
"\n",
" \n",
"**Topics:** Logistic Regression, predict_proba, validation, thresholds, coefficients, actions\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "d4da9f59",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- reserve separate training, validation, and test partitions;\n",
"- convert Logistic Regression probabilities into decisions;\n",
"- choose a threshold using validation data only;\n",
"- measure the threshold trade-off between false positives and false negatives;\n",
"- interpret influential terms through model coefficients;\n",
"- map model scores to allow, analyst-review, and quarantine actions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9579de3e",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"THRESHOLDS = np.arange(0.20, 0.81, 0.05)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "ec160895",
"metadata": {},
"source": [
"## 1. Prepare train, validation, and test data\n",
"\n",
"Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5880993f",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "da9d2bae",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: split the data into 60% training and 40% temporary data.\n",
"X_train, X_temp, y_train, y_temp = None, None, None, None\n",
"\n",
"# TODO 2: split the temporary data equally into validation and test sets.\n",
"X_valid, X_test, y_valid, y_test = None, None, None, None\n",
"\n",
"print(\"Train / validation / test:\", None)\n",
"print(\"Spam rates:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "91f38f11",
"metadata": {},
"source": [
"## 2. Train a probabilistic text classifier\n",
"\n",
"Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fbd38eaa",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create and fit a TF-IDF + Logistic Regression pipeline.\n",
"lr_pipeline = None\n",
"\n",
"# TODO 4: calculate validation spam probabilities.\n",
"valid_probabilities = None\n",
"\n",
"print(\"First probabilities:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "cdd0579e",
"metadata": {},
"source": [
"## 3. Select a threshold on validation data\n",
"\n",
"The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "042a6426",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: evaluate every threshold in THRESHOLDS on validation data.\n",
"threshold_table = None\n",
"\n",
"# TODO 6: among thresholds with spam recall >= 0.80, select the one\n",
"# with the highest precision. If none qualify, use the highest F1.\n",
"selected_threshold = None\n",
"\n",
"print(threshold_table.round(4))\n",
"print(\"Selected threshold:\", selected_threshold)\n"
]
},
{
"cell_type": "markdown",
"id": "0f33c692",
"metadata": {},
"source": [
"## 4. Evaluate once on the untouched test set\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f7567738",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: evaluate the selected threshold once on the untouched test set.\n",
"test_probabilities = None\n",
"test_predictions = None\n",
"test_cm = None\n",
"test_metrics = None\n",
"\n",
"print(pd.Series(test_metrics).round(4))\n",
"print(test_cm)\n"
]
},
{
"cell_type": "markdown",
"id": "fffd1f1c",
"metadata": {},
"source": [
"## 5. Interpret influential terms\n",
"\n",
"Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d8996ab6",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: extract the fitted vocabulary and Logistic Regression coefficients.\n",
"feature_names = None\n",
"coefficients = None\n",
"\n",
"# TODO 9: create a table with the 15 strongest ham terms and\n",
"# the 15 strongest spam terms.\n",
"top_features = None\n",
"top_features\n"
]
},
{
"cell_type": "markdown",
"id": "42879342",
"metadata": {},
"source": [
"## 6. Map probabilities to actions\n",
"\n",
"Use fixed demonstration bands: below 0.30 is `allow`, 0.30–0.70 is `analyst_review`, and 0.70 or above is `quarantine`. A production policy would set these boundaries from business cost, capacity, and validation evidence.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7031034a",
"metadata": {},
"outputs": [],
"source": [
"# TODO 10: map test probabilities to allow, analyst_review, or quarantine.\n",
"policy_actions = None\n",
"action_summary = None\n",
"\n",
"action_summary\n"
]
},
{
"cell_type": "markdown",
"id": "b3545892",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f99dce7",
"metadata": {},
"outputs": [],
"source": [
"assert len(X_train) + len(X_valid) + len(X_test) == len(df)\n",
"assert set(lr_pipeline.named_steps) == {\"tfidf\", \"model\"}\n",
"assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))\n",
"assert threshold_table.index.is_monotonic_increasing\n",
"assert selected_threshold in threshold_table.index\n",
"assert len(test_predictions) == len(y_test)\n",
"assert test_cm.shape == (2, 2)\n",
"assert len(top_features) == 30\n",
"assert set(top_features[\"direction\"]) == {\"ham\", \"spam\"}\n",
"assert set(policy_actions.astype(\"string\").unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"print(\"✓ Lab 5 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "913d3185",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is a validation set needed when selecting a threshold?\n",
"2. What usually happens to spam recall when the threshold is lowered?\n",
"3. Why are large positive coefficients not proof that a term causes spam?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/06_Capstone_SMS_Spam_Detector_Lab.ipynb
+354
View File
@@ -0,0 +1,354 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "59b46ce9",
"metadata": {},
"source": [
"# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n",
"\n",
"\n",
"**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution.\n"
]
},
{
"cell_type": "markdown",
"id": "06204436",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build one complete, reproducible, and leakage-safe spam-detection workflow:\n",
"\n",
"1. load and audit the UCI SMS Spam Collection;\n",
"2. create one stratified holdout test set;\n",
"3. train a word-level TF-IDF and Logistic Regression pipeline;\n",
"4. evaluate spam-sensitive metrics and operational errors;\n",
"5. inspect influential terms and test custom messages;\n",
"6. export evidence and an analyst-facing summary.\n",
"\n",
"The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8afc13cc",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import textwrap\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay,\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"RANDOM_STATE = 42\n",
"DECISION_THRESHOLD = 0.50\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n",
"\n",
"OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n"
]
},
{
"cell_type": "markdown",
"id": "0bf5bfb8",
"metadata": {},
"source": [
"## 1. Load, audit, and split the data\n",
"\n",
"Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5092a6b4",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the UCI file, normalize the two columns, remove invalid\n",
"# rows and conflicts, remove exact duplicates, and create target.\n",
"raw_df = None\n",
"clean_df = None\n",
"\n",
"# TODO 2: create X and y, then make a stratified 80/20 split.\n",
"X = None\n",
"y = None\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"print(\"Raw rows:\", None)\n",
"print(\"Clean rows:\", None)\n",
"print(\"Training / test:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "649bf063",
"metadata": {},
"source": [
"## 2. Build and train the final pipeline\n",
"\n",
"The pipeline learns its vocabulary and IDF weights from the training messages only.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "78a6493b",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create a word unigram+bigram TF-IDF and Logistic Regression pipeline.\n",
"final_pipeline = None\n",
"\n",
"# TODO 4: fit on training data and calculate test probabilities and predictions.\n",
"test_probabilities = None\n",
"test_predictions = None\n",
"\n",
"print(\"Predictions created:\", None)\n"
]
},
{
"cell_type": "markdown",
"id": "b99770cb",
"metadata": {},
"source": [
"## 3. Evaluate the held-out test set\n",
"\n",
"Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ed40674",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: calculate the confusion matrix and capstone metrics.\n",
"cm = None\n",
"tn, fp, fn, tp = None, None, None, None\n",
"metrics = None\n",
"\n",
"print(pd.Series(metrics).round(4))\n",
"print(None)\n"
]
},
{
"cell_type": "markdown",
"id": "3131b3f0",
"metadata": {},
"source": [
"## 4. Create an analyst error table\n",
"\n",
"Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "95449797",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: create prediction_table with message, actual_label,\n",
"# predicted_label, spam_probability, action, and error_type.\n",
"prediction_table = None\n",
"\n",
"# TODO 7: keep only false positives and false negatives in error_table.\n",
"error_table = None\n",
"\n",
"print(\"Errors by type:\")\n",
"print(None)\n",
"error_table.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "e075f283",
"metadata": {},
"source": [
"## 5. Inspect influential words and phrases\n",
"\n",
"Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e8da0477",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: extract the 20 strongest spam terms and 20 strongest ham terms.\n",
"top_terms = None\n",
"top_terms.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "2ecdeca9",
"metadata": {},
"source": [
"## 6. Challenge the detector with new messages\n",
"\n",
"The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcb531ea",
"metadata": {},
"outputs": [],
"source": [
"custom_messages = pd.Series([\n",
" \"Can we move the project meeting to 3 pm?\",\n",
" \"Congratulations! Claim your free cash prize now\",\n",
" \"URGENT: verify your account at the link below\",\n",
" \"You are a w1nner, reply FR33 to collect your reward\",\n",
" \"Please call me when you arrive at the station\",\n",
"], name=\"message\")\n",
"\n",
"# TODO 9: score the custom messages and assign the three policy actions.\n",
"custom_probabilities = None\n",
"custom_results = None\n",
"custom_results\n"
]
},
{
"cell_type": "markdown",
"id": "99e5489b",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f0d90df9",
"metadata": {},
"outputs": [],
"source": [
"# TODO 10: save metrics, predictions, top terms, and a confusion-matrix chart.\n",
"# TODO 11: write an analyst summary containing the dataset size, spam rate,\n",
"# threshold, four primary metrics, FP/FN counts, and one limitation.\n",
"summary = None\n",
"\n",
"print(summary)\n"
]
},
{
"cell_type": "markdown",
"id": "b8fac56b",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1694158b",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_predictions.csv\",\n",
" \"capstone_top_terms.csv\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_summary.txt\",\n",
"}\n",
"\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert len(test_predictions) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert len(prediction_table) == len(y_test)\n",
"assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"assert len(top_terms) == 40\n",
"assert len(custom_results) == 5\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"Limitation\" in summary\n",
"print(\"✓ Lab 6 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "14123800",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. What do the false positives and false negatives mean to a security team?\n",
"2. Why should the custom obfuscated message be included in the capstone?\n",
"3. What evidence is still missing before this classroom model could support real email security decisions?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "60c6f317-85da-455b-80bb-b9a2d6f8b556",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4043e40-4d92-48a4-b5f1-51fe55f2f75b",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Practical/data/SMSSpamCollection
+5574
View File
File diff suppressed because it is too large Load Diff
Lab-4/Lab-4-Practical/data/sms+spam+collection.zip
BIN
View File
Binary file not shown.
Lab-4/Lab-4-Solution.zip
BIN
View File
Binary file not shown.
Lab-4/Lab-4-Solution/01_SMS_Dataset_Inspection_Solution.ipynb
+252
View File
@@ -0,0 +1,252 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "979aed09",
"metadata": {},
"source": [
"# Module 4 — Lab 1: Inspecting the SMS Spam Dataset\n",
"\n",
"\n",
"**Topics:** text data, duplicates, conflicting labels, class distribution\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "c9b13285",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load the original UCI tab-separated text file;\n",
"- audit missing values, exact duplicates, and conflicting labels;\n",
"- create a binary target without modifying the message text;\n",
"- describe class imbalance and message-length differences;\n",
"- explain why aggressive cleaning may remove useful security signals.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8c2b2a4a",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import pandas as pd\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download \"\n",
" \"and extract the official UCI SMS Spam Collection.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "184f1a06",
"metadata": {},
"source": [
"## 1. Load the original collection\n",
"\n",
"The source file has no header. Each row contains a label, one tab character, and the message text.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f376803d",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(\n",
" DATA_PATH,\n",
" sep=\"\\t\",\n",
" header=None,\n",
" names=[\"label\", \"message\"],\n",
" encoding=\"utf-8\",\n",
")\n",
"\n",
"print(\"Raw shape:\", raw_df.shape)\n",
"print(raw_df[\"label\"].value_counts(dropna=False))\n",
"raw_df.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "4ae84500",
"metadata": {},
"source": [
"## 2. Audit the records\n",
"\n",
"Exact duplicates can cross a random train-test split and make evaluation too optimistic. A more serious problem occurs when identical text has different labels. Audit both conditions before modeling.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "177c72a1",
"metadata": {},
"outputs": [],
"source": [
"audited_df = raw_df.copy()\n",
"audited_df[\"label\"] = audited_df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"audited_df[\"message\"] = audited_df[\"message\"].astype(\"string\").str.strip()\n",
"audited_df = audited_df.dropna(subset=[\"label\", \"message\"])\n",
"audited_df = audited_df[audited_df[\"message\"].ne(\"\")]\n",
"audited_df = audited_df[audited_df[\"label\"].isin([\"ham\", \"spam\"])]\n",
"\n",
"exact_duplicate_count = int(audited_df.duplicated(subset=[\"label\", \"message\"]).sum())\n",
"label_counts_per_message = audited_df.groupby(\"message\")[\"label\"].nunique()\n",
"conflicting_messages = label_counts_per_message[label_counts_per_message.gt(1)].index\n",
"\n",
"print(\"Exact duplicate rows:\", exact_duplicate_count)\n",
"print(\"Conflicting message texts:\", len(conflicting_messages))\n"
]
},
{
"cell_type": "markdown",
"id": "b9d41a33",
"metadata": {},
"source": [
"## 3. Build the clean classroom table\n",
"\n",
"Remove ambiguous messages before exact duplicates, then map `ham → 0` and `spam → 1`. Keep the original text intact.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b0b915b1",
"metadata": {},
"outputs": [],
"source": [
"clean_df = audited_df[~audited_df[\"message\"].isin(conflicting_messages)].copy()\n",
"clean_df = clean_df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"clean_df[\"target\"] = clean_df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"print(\"Clean shape:\", clean_df.shape)\n",
"print(clean_df[\"target\"].value_counts().sort_index().rename(index={0: \"ham\", 1: \"spam\"}))\n"
]
},
{
"cell_type": "markdown",
"id": "74762f79",
"metadata": {},
"source": [
"## 4. Describe the classes and message lengths\n",
"\n",
"Length is used only for exploration. The later models will learn from the message text itself.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1334e9da",
"metadata": {},
"outputs": [],
"source": [
"clean_df[\"character_count\"] = clean_df[\"message\"].str.len()\n",
"clean_df[\"word_count\"] = clean_df[\"message\"].str.split().str.len()\n",
"\n",
"length_summary = clean_df.groupby(\"label\").agg(\n",
" message_count=(\"message\", \"size\"),\n",
" median_characters=(\"character_count\", \"median\"),\n",
" median_words=(\"word_count\", \"median\"),\n",
")\n",
"class_distribution = clean_df[\"label\"].value_counts(normalize=True)\n",
"\n",
"print(length_summary)\n",
"print(\"Class proportions:\")\n",
"print(class_distribution)\n"
]
},
{
"cell_type": "markdown",
"id": "34c637b9",
"metadata": {},
"source": [
"## Security note: do not over-clean\n",
"\n",
"URLs, phone numbers, currency symbols, punctuation, unusual capitalization, and deliberate misspellings may contain spam evidence. Begin with the vectorizer defaults. Add normalization only when an experiment demonstrates that it improves validation performance without hiding operationally important signals.\n"
]
},
{
"cell_type": "markdown",
"id": "27f4bb63",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "870a2fbe",
"metadata": {},
"outputs": [],
"source": [
"assert list(raw_df.columns) == [\"label\", \"message\"]\n",
"assert clean_df[\"message\"].notna().all()\n",
"assert clean_df[\"message\"].ne(\"\").all()\n",
"assert set(clean_df[\"label\"].unique()) == {\"ham\", \"spam\"}\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert not clean_df[\"message\"].isin(conflicting_messages).any()\n",
"assert class_distribution.loc[\"ham\"] > class_distribution.loc[\"spam\"]\n",
"assert {\"character_count\", \"word_count\"}.issubset(clean_df.columns)\n",
"print(\"✓ Lab 1 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "3a766ad5",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why should exact duplicates be removed before the train-test split?** \n",
" A duplicate can otherwise appear in both partitions, allowing the model to receive nearly identical training and test evidence and inflating the reported score.\n",
"2. **Why might removing URLs and currency symbols reduce detection quality?** \n",
" Those tokens often carry direct spam intent. Removing them can discard discriminative evidence rather than remove harmless noise.\n",
"3. **Why is accuracy alone risky for this dataset?** \n",
" The ham class is much larger, so a model can obtain high accuracy while missing many spam messages. Spam precision, recall, F1, and the confusion matrix are also required.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcda6cf6-3ba4-4229-8991-8efb2209d294",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/02_Bag_of_Words_TFIDF_Solution.ipynb
+273
View File
@@ -0,0 +1,273 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "942ac4eb",
"metadata": {},
"source": [
"# Module 4 — Lab 2: Bag of Words and TF-IDF\n",
"\n",
"\n",
"**Topics:** CountVectorizer, TfidfVectorizer, sparse matrices, n-grams, leakage\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "554dc6ef",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- explain how Bag of Words maps documents to numeric vectors;\n",
"- inspect a document-term matrix and its vocabulary;\n",
"- fit TF-IDF on training text and transform unseen text;\n",
"- describe why text matrices are high-dimensional and sparse;\n",
"- compare unigram features with unigram-plus-bigram features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dee543fe",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "637b0eaa",
"metadata": {},
"source": [
"## 1. Make Bag of Words visible\n",
"\n",
"Use three short messages so that every row, token, and count can be inspected directly.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c3d5152",
"metadata": {},
"outputs": [],
"source": [
"tiny_messages = [\n",
" \"free prize now\",\n",
" \"free meeting today\",\n",
" \"project meeting today\",\n",
"]\n",
"\n",
"count_vectorizer = CountVectorizer()\n",
"tiny_bow = count_vectorizer.fit_transform(tiny_messages)\n",
"tiny_bow_df = pd.DataFrame(\n",
" tiny_bow.toarray(),\n",
" columns=count_vectorizer.get_feature_names_out(),\n",
" index=[\"message_1\", \"message_2\", \"message_3\"],\n",
")\n",
"tiny_bow_df\n"
]
},
{
"cell_type": "markdown",
"id": "e3e98734",
"metadata": {},
"source": [
"## 2. Prepare the UCI messages\n",
"\n",
"Remove exact duplicates before splitting. Do not fit a vocabulary on the complete dataset because the held-out test messages must remain unseen.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9841f9b7",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"\n",
"conflicts = df.groupby(\"message\")[\"label\"].nunique()\n",
"conflict_texts = conflicts[conflicts.gt(1)].index\n",
"df = df[~df[\"message\"].isin(conflict_texts)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"print(\"Clean messages:\", len(df))\n"
]
},
{
"cell_type": "markdown",
"id": "92848cea",
"metadata": {},
"source": [
"## 3. Fit TF-IDF without leakage\n",
"\n",
"`fit_transform` learns a vocabulary and inverse-document-frequency weights from training messages. The test set receives only `transform`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e5a89189",
"metadata": {},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.20,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"\n",
"tfidf_vectorizer = TfidfVectorizer(lowercase=True)\n",
"X_train_tfidf = tfidf_vectorizer.fit_transform(X_train)\n",
"X_test_tfidf = tfidf_vectorizer.transform(X_test)\n",
"\n",
"density = X_train_tfidf.nnz / (X_train_tfidf.shape[0] * X_train_tfidf.shape[1])\n",
"print(\"Training matrix shape:\", X_train_tfidf.shape)\n",
"print(\"Test matrix shape:\", X_test_tfidf.shape)\n",
"print(\"Training density:\", f\"{density:.4%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "a4230176",
"metadata": {},
"source": [
"## 4. Inspect the learned feature space\n",
"\n",
"A high mean TF-IDF value does not automatically mean a term is a spam indicator. This step only describes the training corpus. A classifier will learn which terms distinguish the classes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "90e679d0",
"metadata": {},
"outputs": [],
"source": [
"mean_tfidf = np.asarray(X_train_tfidf.mean(axis=0)).ravel()\n",
"top_terms = pd.Series(\n",
" mean_tfidf,\n",
" index=tfidf_vectorizer.get_feature_names_out(),\n",
" name=\"mean_tfidf\",\n",
").sort_values(ascending=False).head(15)\n",
"top_terms\n"
]
},
{
"cell_type": "markdown",
"id": "2ebe7eee",
"metadata": {},
"source": [
"## 5. Add short phrases with bigrams\n",
"\n",
"A unigram represents one token. A bigram can preserve short phrases such as `call now` or `free entry`, but it also increases the number of features.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c1de19b",
"metadata": {},
"outputs": [],
"source": [
"ngram_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=1)\n",
"X_train_ngram = ngram_vectorizer.fit_transform(X_train)\n",
"ngram_features = ngram_vectorizer.get_feature_names_out()\n",
"example_bigrams = [term for term in ngram_features if \" \" in term][:10]\n",
"\n",
"print(\"Unigram features:\", X_train_tfidf.shape[1])\n",
"print(\"Unigram + bigram features:\", X_train_ngram.shape[1])\n",
"print(\"Example bigrams:\", example_bigrams)\n"
]
},
{
"cell_type": "markdown",
"id": "64432265",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "36f18994",
"metadata": {},
"outputs": [],
"source": [
"assert tiny_bow.shape == (3, len(count_vectorizer.get_feature_names_out()))\n",
"assert tiny_bow_df.loc[\"message_1\", \"free\"] == 1\n",
"assert len(X_train) + len(X_test) == len(df)\n",
"assert X_train_tfidf.shape[0] == len(X_train)\n",
"assert X_test_tfidf.shape[0] == len(X_test)\n",
"assert X_train_tfidf.shape[1] == X_test_tfidf.shape[1]\n",
"assert X_train_tfidf.nnz < X_train_tfidf.shape[0] * X_train_tfidf.shape[1]\n",
"assert X_train_ngram.shape[1] >= X_train_tfidf.shape[1]\n",
"assert len(top_terms) == 15\n",
"print(\"✓ Lab 2 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "991fa410",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is a text feature matrix usually sparse?** \n",
" Each message uses only a small fraction of the complete vocabulary, so most document-term cells are zero.\n",
"2. **Why must the vectorizer be fitted only on training messages?** \n",
" Fitting on all messages reveals test-set vocabulary and frequency information during training, which is data leakage and can make evaluation optimistic.\n",
"3. **What trade-off is introduced by adding bigrams?** \n",
" Bigrams preserve useful short phrases but expand the feature space, consume more memory, and can overfit when phrases are rare.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/03_Naive_Bayes_Baseline_Solution.ipynb
+294
View File
@@ -0,0 +1,294 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "14ac1890",
"metadata": {},
"source": [
"# Module 4 — Lab 3: A Leakage-Safe Naive Bayes Baseline\n",
"\n",
"**Topics:** Pipeline, MultinomialNB, confusion matrix, baseline, error analysis\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "af2dc7be",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- place TF-IDF and Multinomial Naive Bayes in one pipeline;\n",
"- explain how the pipeline prevents preprocessing leakage;\n",
"- calculate spam precision, recall, and F1;\n",
"- compare a trained classifier with a naive baseline;\n",
"- inspect false positives and false negatives.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bced8aa9",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "aa622049",
"metadata": {},
"source": [
"## 1. Load, audit, and split the messages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9efd4a80",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n",
"print(\"Training messages:\", len(X_train))\n",
"print(\"Test messages:\", len(X_test))\n",
"print(\"Training spam rate:\", f\"{y_train.mean():.2%}\")\n",
"print(\"Test spam rate:\", f\"{y_test.mean():.2%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "263fa9cf",
"metadata": {},
"source": [
"## 2. Build and train one pipeline\n",
"\n",
"The pipeline fits TF-IDF only on `X_train`. During prediction, it applies the already-fitted transformation to `X_test`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aed27122",
"metadata": {},
"outputs": [],
"source": [
"nb_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(lowercase=True, ngram_range=(1, 1))),\n",
" (\"model\", MultinomialNB(alpha=1.0)),\n",
"])\n",
"nb_pipeline.fit(X_train, y_train)\n",
"y_pred = nb_pipeline.predict(X_test)\n",
"\n",
"print(nb_pipeline)\n"
]
},
{
"cell_type": "markdown",
"id": "50d5bc04",
"metadata": {},
"source": [
"## 3. Evaluate the spam class\n",
"\n",
"- **Spam precision:** when the model blocks a message, how often is it truly spam?\n",
"- **Spam recall:** of all spam messages, how many did the model catch?\n",
"- **Spam F1:** harmonic mean of spam precision and recall.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32b32feb",
"metadata": {},
"outputs": [],
"source": [
"cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
"tn, fp, fn, tp = cm.ravel()\n",
"\n",
"nb_metrics = {\n",
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
"}\n",
"\n",
"print(pd.Series(nb_metrics).round(4))\n",
"print(\"\\nClassification report:\")\n",
"print(classification_report(y_test, y_pred, target_names=[\"ham\", \"spam\"], zero_division=0))\n"
]
},
{
"cell_type": "markdown",
"id": "3a7568ba",
"metadata": {},
"source": [
"## 4. Compare with a naive baseline\n",
"\n",
"Because ham is the majority class, a model that always predicts ham may achieve deceptively high accuracy while detecting no spam.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7198e0cf",
"metadata": {},
"outputs": [],
"source": [
"dummy = DummyClassifier(strategy=\"most_frequent\")\n",
"dummy.fit(X_train.to_frame(), y_train)\n",
"dummy_pred = dummy.predict(X_test.to_frame())\n",
"\n",
"comparison = pd.DataFrame([\n",
" {\n",
" \"model\": \"Most-frequent baseline\",\n",
" \"accuracy\": accuracy_score(y_test, dummy_pred),\n",
" \"spam_f1\": f1_score(y_test, dummy_pred, pos_label=1, zero_division=0),\n",
" },\n",
" {\n",
" \"model\": \"TF-IDF + MultinomialNB\",\n",
" \"accuracy\": nb_metrics[\"accuracy\"],\n",
" \"spam_f1\": nb_metrics[\"spam_f1\"],\n",
" },\n",
"]).set_index(\"model\")\n",
"comparison\n"
]
},
{
"cell_type": "markdown",
"id": "56233ccf",
"metadata": {},
"source": [
"## 5. Inspect operational errors\n",
"\n",
"A false positive blocks a legitimate message. A false negative allows spam through. Both deserve inspection rather than being hidden inside one score.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e601a648",
"metadata": {},
"outputs": [],
"source": [
"error_analysis = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual\": y_test.to_numpy(),\n",
" \"predicted\": y_pred,\n",
"})\n",
"errors = error_analysis[error_analysis[\"actual\"].ne(error_analysis[\"predicted\"])].copy()\n",
"errors[\"error_type\"] = np.where(\n",
" (errors[\"actual\"] == 0) & (errors[\"predicted\"] == 1),\n",
" \"false_positive\",\n",
" \"false_negative\",\n",
")\n",
"\n",
"print(errors[\"error_type\"].value_counts())\n",
"errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "a818ba0b",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "72c65672",
"metadata": {},
"outputs": [],
"source": [
"assert list(nb_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert isinstance(nb_pipeline.named_steps[\"model\"], MultinomialNB)\n",
"assert len(y_pred) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert set(nb_metrics) == {\"accuracy\", \"spam_precision\", \"spam_recall\", \"spam_f1\"}\n",
"assert comparison.loc[\"Most-frequent baseline\", \"spam_f1\"] == 0\n",
"assert set(errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 3 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "7f0d0753",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is the most-frequent classifier an important baseline?** \n",
" It reveals how much accuracy comes from class imbalance alone. A useful spam detector must outperform the baseline on spam-sensitive metrics.\n",
"2. **Which operational harm is represented by a false positive?** \n",
" A legitimate message is incorrectly blocked or quarantined, which can interrupt communication and reduce user trust.\n",
"3. **Why should TF-IDF and the model be placed in one pipeline?** \n",
" The pipeline guarantees that vocabulary and IDF statistics are learned only during fitting and then reused consistently, reducing leakage and preprocessing mistakes.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/04_Linear_Model_Comparison_Solution.ipynb
+304
View File
@@ -0,0 +1,304 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "d43eebb6",
"metadata": {},
"source": [
"# Module 4 — Lab 4: Comparing Linear Text Classifiers\n",
"\n",
"\n",
"**Topics:** Naive Bayes, Perceptron, Linear SVM, Logistic Regression, fair comparison\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "78892370",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare several classifiers on one fixed train-test split;\n",
"- explain why linear decision boundaries work well with sparse TF-IDF features;\n",
"- choose a model using spam-sensitive metrics instead of accuracy alone;\n",
"- connect false positives and false negatives to operational consequences;\n",
"- explain why kernel SVM is not required for this classroom workflow.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d9e93987",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression, Perceptron\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.naive_bayes import MultinomialNB\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.svm import LinearSVC\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "eae4780b",
"metadata": {},
"source": [
"## 1. Prepare one fixed experiment\n",
"\n",
"Every model receives exactly the same training and test messages. Each pipeline contains its own vectorizer so that no test information is used to create the vocabulary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6d147533",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, random_state=RANDOM_STATE, stratify=y\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "899ce6a8",
"metadata": {},
"source": [
"## 2. Define the candidate pipelines\n",
"\n",
"All models use unigram and bigram TF-IDF features. This controls the representation so that the comparison focuses on the learning algorithm.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7ac547e0",
"metadata": {},
"outputs": [],
"source": [
"def make_pipeline(model):\n",
" return Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", model),\n",
" ])\n",
"\n",
"\n",
"models = {\n",
" \"MultinomialNB\": make_pipeline(MultinomialNB(alpha=1.0)),\n",
" \"Perceptron\": make_pipeline(Perceptron(\n",
" max_iter=2000,\n",
" tol=1e-3,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
" \"LinearSVC\": make_pipeline(LinearSVC(random_state=RANDOM_STATE)),\n",
" \"LogisticRegression\": make_pipeline(LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"}\n",
"\n",
"print(\"Models:\", list(models))\n"
]
},
{
"cell_type": "markdown",
"id": "ebbc5289",
"metadata": {},
"source": [
"## 3. Train and evaluate fairly\n",
"\n",
"Linear models are a practical match for high-dimensional sparse text. A kernel SVM can be computationally expensive and does not add a necessary learning outcome here, so the experiment uses `LinearSVC`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc716625",
"metadata": {},
"outputs": [],
"source": [
"fitted_models = {}\n",
"predictions = {}\n",
"rows = []\n",
"\n",
"for model_name, pipeline in models.items():\n",
" pipeline.fit(X_train, y_train)\n",
" y_pred = pipeline.predict(X_test)\n",
" cm = confusion_matrix(y_test, y_pred, labels=[0, 1])\n",
" tn, fp, fn, tp = cm.ravel()\n",
"\n",
" fitted_models[model_name] = pipeline\n",
" predictions[model_name] = y_pred\n",
" rows.append({\n",
" \"model\": model_name,\n",
" \"accuracy\": accuracy_score(y_test, y_pred),\n",
" \"spam_precision\": precision_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, y_pred, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" })\n",
"\n",
"comparison = (\n",
" pd.DataFrame(rows)\n",
" .set_index(\"model\")\n",
" .sort_values([\"spam_f1\", \"spam_recall\"], ascending=False)\n",
")\n",
"comparison.round(4)\n"
]
},
{
"cell_type": "markdown",
"id": "e6d4d8a6",
"metadata": {},
"source": [
"## 4. Select and inspect the strongest model\n",
"\n",
"For this exercise, use spam F1 as the primary selection score and spam recall as a secondary consideration. A real organization would define the cost of each error first.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "86f3ebb4",
"metadata": {},
"outputs": [],
"source": [
"best_model_name = comparison[\"spam_f1\"].idxmax()\n",
"best_pipeline = fitted_models[best_model_name]\n",
"best_predictions = predictions[best_model_name]\n",
"\n",
"best_errors = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual\": y_test.to_numpy(),\n",
" \"predicted\": best_predictions,\n",
"})\n",
"best_errors = best_errors[best_errors[\"actual\"].ne(best_errors[\"predicted\"])].copy()\n",
"best_errors[\"error_type\"] = np.where(\n",
" (best_errors[\"actual\"] == 0) & (best_errors[\"predicted\"] == 1),\n",
" \"false_positive\",\n",
" \"false_negative\",\n",
")\n",
"\n",
"print(\"Best model by spam F1:\", best_model_name)\n",
"print(\"Number of errors:\", len(best_errors))\n",
"best_errors.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "a6dbf81d",
"metadata": {},
"source": [
"## Interpretation checkpoint\n",
"\n",
"A higher score does not prove that a model is universally better. The result applies to this split, this dataset, and this feature configuration. Lab 5 adds probability thresholds so that an analyst can change the decision policy without training a new model.\n"
]
},
{
"cell_type": "markdown",
"id": "a4c51d44",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bd14558",
"metadata": {},
"outputs": [],
"source": [
"assert set(models) == {\"MultinomialNB\", \"Perceptron\", \"LinearSVC\", \"LogisticRegression\"}\n",
"assert set(fitted_models) == set(models)\n",
"assert set(predictions) == set(models)\n",
"assert comparison.shape == (4, 6)\n",
"assert comparison.index.is_unique\n",
"assert comparison[\"spam_f1\"].between(0, 1).all()\n",
"assert best_model_name in models\n",
"assert len(best_predictions) == len(y_test)\n",
"assert set(best_errors[\"error_type\"].unique()).issubset({\"false_positive\", \"false_negative\"})\n",
"print(\"✓ Lab 4 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "19c2b755",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why must every model use the same data split?** \n",
" Changing both the split and model would confound the comparison. A fixed split isolates the effect of the classifier and makes the scores directly comparable.\n",
"2. **Why is a linear SVM appropriate for TF-IDF features?** \n",
" TF-IDF creates many sparse features. Linear SVM can learn a separating hyperplane efficiently in this high-dimensional space without calculating an expensive nonlinear kernel matrix.\n",
"3. **Why might the highest-accuracy model still be unsuitable for spam filtering?** \n",
" Accuracy can be dominated by the majority ham class. A model may still miss too much spam or block too many legitimate messages, which is visible in spam recall, precision, and error counts.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/05_Probability_Thresholds_Interpretation_Solution.ipynb
+362
View File
@@ -0,0 +1,362 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b438e257",
"metadata": {},
"source": [
"# Module 4 — Lab 5: Probability, Thresholds, and Interpretation\n",
"\n",
"\n",
"**Topics:** Logistic Regression, predict_proba, validation, thresholds, coefficients, actions\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "e52f8f6f",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- reserve separate training, validation, and test partitions;\n",
"- convert Logistic Regression probabilities into decisions;\n",
"- choose a threshold using validation data only;\n",
"- measure the threshold trade-off between false positives and false negatives;\n",
"- interpret influential terms through model coefficients;\n",
"- map model scores to allow, analyst-review, and quarantine actions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "840495c7",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" accuracy_score,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"RANDOM_STATE = 42\n",
"THRESHOLDS = np.arange(0.20, 0.81, 0.05)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n"
]
},
{
"cell_type": "markdown",
"id": "a4aa5a16",
"metadata": {},
"source": [
"## 1. Prepare train, validation, and test data\n",
"\n",
"Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47bfe88f",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"df[\"label\"] = df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"df[\"message\"] = df[\"message\"].astype(\"string\").str.strip()\n",
"df = df.dropna(subset=[\"label\", \"message\"])\n",
"df = df[df[\"label\"].isin([\"ham\", \"spam\"]) & df[\"message\"].ne(\"\")]\n",
"conflict_counts = df.groupby(\"message\")[\"label\"].nunique()\n",
"df = df[~df[\"message\"].isin(conflict_counts[conflict_counts.gt(1)].index)]\n",
"df = df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"\n",
"X = df[\"message\"]\n",
"y = df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cf3ba33b",
"metadata": {},
"outputs": [],
"source": [
"X_train, X_temp, y_train, y_temp = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.40,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"X_valid, X_test, y_valid, y_test = train_test_split(\n",
" X_temp,\n",
" y_temp,\n",
" test_size=0.50,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y_temp,\n",
")\n",
"\n",
"print(\"Train / validation / test:\", len(X_train), len(X_valid), len(X_test))\n",
"print(\"Spam rates:\", f\"{y_train.mean():.2%}\", f\"{y_valid.mean():.2%}\", f\"{y_test.mean():.2%}\")\n"
]
},
{
"cell_type": "markdown",
"id": "82a1cadd",
"metadata": {},
"source": [
"## 2. Train a probabilistic text classifier\n",
"\n",
"Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ecb39196",
"metadata": {},
"outputs": [],
"source": [
"lr_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"])\n",
"lr_pipeline.fit(X_train, y_train)\n",
"valid_probabilities = lr_pipeline.predict_proba(X_valid)[:, 1]\n",
"\n",
"print(\"First probabilities:\", np.round(valid_probabilities[:10], 3))\n"
]
},
{
"cell_type": "markdown",
"id": "b2e41b60",
"metadata": {},
"source": [
"## 3. Select a threshold on validation data\n",
"\n",
"The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "54cf81c8",
"metadata": {},
"outputs": [],
"source": [
"rows = []\n",
"for threshold in THRESHOLDS:\n",
" predictions = (valid_probabilities >= threshold).astype(int)\n",
" cm = confusion_matrix(y_valid, predictions, labels=[0, 1])\n",
" tn, fp, fn, tp = cm.ravel()\n",
" rows.append({\n",
" \"threshold\": threshold,\n",
" \"spam_precision\": precision_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_valid, predictions, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" })\n",
"\n",
"threshold_table = pd.DataFrame(rows).set_index(\"threshold\")\n",
"eligible = threshold_table[threshold_table[\"spam_recall\"].ge(0.80)]\n",
"if eligible.empty:\n",
" selected_threshold = float(threshold_table[\"spam_f1\"].idxmax())\n",
"else:\n",
" selected_threshold = float(\n",
" eligible.sort_values([\"spam_precision\", \"spam_f1\"], ascending=False).index[0]\n",
" )\n",
"\n",
"print(threshold_table.round(4))\n",
"print(\"Selected threshold:\", selected_threshold)\n"
]
},
{
"cell_type": "markdown",
"id": "b1658a9c",
"metadata": {},
"source": [
"## 4. Evaluate once on the untouched test set\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f5d43c65",
"metadata": {},
"outputs": [],
"source": [
"test_probabilities = lr_pipeline.predict_proba(X_test)[:, 1]\n",
"test_predictions = (test_probabilities >= selected_threshold).astype(int)\n",
"test_cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n",
"tn, fp, fn, tp = test_cm.ravel()\n",
"test_metrics = {\n",
" \"threshold\": selected_threshold,\n",
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
" \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
"}\n",
"\n",
"print(pd.Series(test_metrics).round(4))\n",
"print(test_cm)\n"
]
},
{
"cell_type": "markdown",
"id": "f41cbbb5",
"metadata": {},
"source": [
"## 5. Interpret influential terms\n",
"\n",
"Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "87ea0bc0",
"metadata": {},
"outputs": [],
"source": [
"feature_names = lr_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n",
"coefficients = lr_pipeline.named_steps[\"model\"].coef_[0]\n",
"coefficient_series = pd.Series(coefficients, index=feature_names)\n",
"\n",
"top_ham = coefficient_series.nsmallest(15).rename(\"coefficient\").reset_index()\n",
"top_ham.columns = [\"term\", \"coefficient\"]\n",
"top_ham[\"direction\"] = \"ham\"\n",
"\n",
"top_spam = coefficient_series.nlargest(15).rename(\"coefficient\").reset_index()\n",
"top_spam.columns = [\"term\", \"coefficient\"]\n",
"top_spam[\"direction\"] = \"spam\"\n",
"\n",
"top_features = pd.concat([top_spam, top_ham], ignore_index=True)\n",
"top_features\n"
]
},
{
"cell_type": "markdown",
"id": "7705ca0e",
"metadata": {},
"source": [
"## 6. Map probabilities to actions\n",
"\n",
"Use fixed demonstration bands: below 0.30 is `allow`, 0.30–0.70 is `analyst_review`, and 0.70 or above is `quarantine`. A production policy would set these boundaries from business cost, capacity, and validation evidence.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9a32d7b4",
"metadata": {},
"outputs": [],
"source": [
"policy_actions = pd.cut(\n",
" test_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
")\n",
"action_summary = pd.crosstab(\n",
" pd.Series(y_test.to_numpy(), name=\"actual\"),\n",
" pd.Series(policy_actions, name=\"action\"),\n",
").rename(index={0: \"ham\", 1: \"spam\"})\n",
"action_summary\n"
]
},
{
"cell_type": "markdown",
"id": "d9e5fb09",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3da1834c",
"metadata": {},
"outputs": [],
"source": [
"assert len(X_train) + len(X_valid) + len(X_test) == len(df)\n",
"assert set(lr_pipeline.named_steps) == {\"tfidf\", \"model\"}\n",
"assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))\n",
"assert threshold_table.index.is_monotonic_increasing\n",
"assert selected_threshold in threshold_table.index\n",
"assert len(test_predictions) == len(y_test)\n",
"assert test_cm.shape == (2, 2)\n",
"assert len(top_features) == 30\n",
"assert set(top_features[\"direction\"]) == {\"ham\", \"spam\"}\n",
"assert set(policy_actions.astype(\"string\").unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"print(\"✓ Lab 5 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "d681ad09",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why is a validation set needed when selecting a threshold?** \n",
" Using test labels to choose the threshold would tune the policy to the final evaluation data. Validation data supports selection while keeping the test estimate independent.\n",
"2. **What usually happens to spam recall when the threshold is lowered?** \n",
" More messages exceed the lower threshold, so spam recall usually rises, but false positives may also increase and precision may fall.\n",
"3. **Why are large positive coefficients not proof that a term causes spam?** \n",
" A coefficient records an association learned from this corpus and representation. It may reflect dataset-specific patterns, confounding, or shortcuts rather than causation.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/06_Capstone_SMS_Spam_Detector_Solution.ipynb
+456
View File
@@ -0,0 +1,456 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4116df5d",
"metadata": {},
"source": [
"# Module 4 — Lab 6: Capstone — An End-to-End SMS Spam Detector\n",
"\n",
" \n",
"**Topics:** complete workflow, Logistic Regression, actions, error analysis, reporting\n",
"\n",
"> **Solution version:** This notebook contains the complete solution. Run the cells in order and compare the results with your Lab notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "81a0abfa",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build one complete, reproducible, and leakage-safe spam-detection workflow:\n",
"\n",
"1. load and audit the UCI SMS Spam Collection;\n",
"2. create one stratified holdout test set;\n",
"3. train a word-level TF-IDF and Logistic Regression pipeline;\n",
"4. evaluate spam-sensitive metrics and operational errors;\n",
"5. inspect influential terms and test custom messages;\n",
"6. export evidence and an analyst-facing summary.\n",
"\n",
"The required model is Logistic Regression because it provides probabilities and interpretable coefficients. Lab 4 already established how to compare it with Naive Bayes, Perceptron, and Linear SVM.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b945ce17",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import textwrap\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay,\n",
" accuracy_score,\n",
" classification_report,\n",
" confusion_matrix,\n",
" f1_score,\n",
" precision_score,\n",
" recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"\n",
"DATA_PATH = Path(\"data/SMSSpamCollection\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"RANDOM_STATE = 42\n",
"DECISION_THRESHOLD = 0.50\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset.\"\n",
" )\n",
"\n",
"OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n"
]
},
{
"cell_type": "markdown",
"id": "8ea5e886",
"metadata": {},
"source": [
"## 1. Load, audit, and split the data\n",
"\n",
"Perform record-level cleaning before splitting so that exact duplicates cannot cross the train-test boundary.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b604f8da",
"metadata": {},
"outputs": [],
"source": [
"raw_df = pd.read_csv(DATA_PATH, sep=\"\\t\", header=None, names=[\"label\", \"message\"])\n",
"clean_df = raw_df.copy()\n",
"clean_df[\"label\"] = clean_df[\"label\"].astype(\"string\").str.strip().str.lower()\n",
"clean_df[\"message\"] = clean_df[\"message\"].astype(\"string\").str.strip()\n",
"clean_df = clean_df.dropna(subset=[\"label\", \"message\"])\n",
"clean_df = clean_df[\n",
" clean_df[\"label\"].isin([\"ham\", \"spam\"]) & clean_df[\"message\"].ne(\"\")\n",
"]\n",
"conflict_counts = clean_df.groupby(\"message\")[\"label\"].nunique()\n",
"conflict_texts = conflict_counts[conflict_counts.gt(1)].index\n",
"clean_df = clean_df[~clean_df[\"message\"].isin(conflict_texts)]\n",
"clean_df = clean_df.drop_duplicates(subset=[\"label\", \"message\"]).reset_index(drop=True)\n",
"clean_df[\"target\"] = clean_df[\"label\"].map({\"ham\": 0, \"spam\": 1}).astype(int)\n",
"\n",
"X = clean_df[\"message\"]\n",
"y = clean_df[\"target\"]\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X,\n",
" y,\n",
" test_size=0.20,\n",
" random_state=RANDOM_STATE,\n",
" stratify=y,\n",
")\n",
"\n",
"print(\"Raw rows:\", len(raw_df))\n",
"print(\"Clean rows:\", len(clean_df))\n",
"print(\"Training / test:\", len(X_train), len(X_test))\n"
]
},
{
"cell_type": "markdown",
"id": "578afb0e",
"metadata": {},
"source": [
"## 2. Build and train the final pipeline\n",
"\n",
"The pipeline learns its vocabulary and IDF weights from the training messages only.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b9eafd46",
"metadata": {},
"outputs": [],
"source": [
"final_pipeline = Pipeline([\n",
" (\"tfidf\", TfidfVectorizer(\n",
" lowercase=True,\n",
" ngram_range=(1, 2),\n",
" min_df=1,\n",
" sublinear_tf=True,\n",
" )),\n",
" (\"model\", LogisticRegression(\n",
" max_iter=2000,\n",
" random_state=RANDOM_STATE,\n",
" )),\n",
"])\n",
"final_pipeline.fit(X_train, y_train)\n",
"test_probabilities = final_pipeline.predict_proba(X_test)[:, 1]\n",
"test_predictions = (test_probabilities >= DECISION_THRESHOLD).astype(int)\n",
"\n",
"print(\"Predictions created:\", len(test_predictions))\n"
]
},
{
"cell_type": "markdown",
"id": "7b6ed943",
"metadata": {},
"source": [
"## 3. Evaluate the held-out test set\n",
"\n",
"Do not change the model or threshold after reading these test results. Additional tuning would require a validation set or cross-validation inside the training partition.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7faddf17",
"metadata": {},
"outputs": [],
"source": [
"cm = confusion_matrix(y_test, test_predictions, labels=[0, 1])\n",
"tn, fp, fn, tp = cm.ravel()\n",
"metrics = {\n",
" \"decision_threshold\": DECISION_THRESHOLD,\n",
" \"accuracy\": accuracy_score(y_test, test_predictions),\n",
" \"spam_precision\": precision_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_recall\": recall_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"spam_f1\": f1_score(y_test, test_predictions, pos_label=1, zero_division=0),\n",
" \"true_negatives\": int(tn),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"true_positives\": int(tp),\n",
"}\n",
"\n",
"print(pd.Series(metrics).round(4))\n",
"print(classification_report(\n",
" y_test,\n",
" test_predictions,\n",
" target_names=[\"ham\", \"spam\"],\n",
" zero_division=0,\n",
"))\n"
]
},
{
"cell_type": "markdown",
"id": "57c486cb",
"metadata": {},
"source": [
"## 4. Create an analyst error table\n",
"\n",
"Sort or filter the exported prediction table later to inspect high-confidence mistakes and messages routed to review.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d612fce7",
"metadata": {},
"outputs": [],
"source": [
"actions = pd.cut(\n",
" test_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
").astype(\"string\")\n",
"\n",
"prediction_table = pd.DataFrame({\n",
" \"message\": X_test.to_numpy(),\n",
" \"actual_label\": np.where(y_test.to_numpy() == 1, \"spam\", \"ham\"),\n",
" \"predicted_label\": np.where(test_predictions == 1, \"spam\", \"ham\"),\n",
" \"spam_probability\": test_probabilities,\n",
" \"action\": actions,\n",
"})\n",
"prediction_table[\"error_type\"] = np.select(\n",
" [\n",
" (prediction_table[\"actual_label\"] == \"ham\") & (prediction_table[\"predicted_label\"] == \"spam\"),\n",
" (prediction_table[\"actual_label\"] == \"spam\") & (prediction_table[\"predicted_label\"] == \"ham\"),\n",
" ],\n",
" [\"false_positive\", \"false_negative\"],\n",
" default=\"correct\",\n",
")\n",
"error_table = prediction_table[prediction_table[\"error_type\"].ne(\"correct\")].copy()\n",
"\n",
"print(\"Errors by type:\")\n",
"print(error_table[\"error_type\"].value_counts())\n",
"error_table.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "99d1bba1",
"metadata": {},
"source": [
"## 5. Inspect influential words and phrases\n",
"\n",
"Coefficients are useful for transparency and shortcut detection, but they do not explain every individual prediction or prove causality.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ebf3fde7",
"metadata": {},
"outputs": [],
"source": [
"feature_names = final_pipeline.named_steps[\"tfidf\"].get_feature_names_out()\n",
"coefficients = pd.Series(\n",
" final_pipeline.named_steps[\"model\"].coef_[0],\n",
" index=feature_names,\n",
" name=\"coefficient\",\n",
")\n",
"\n",
"top_spam = coefficients.nlargest(20).reset_index()\n",
"top_spam.columns = [\"term\", \"coefficient\"]\n",
"top_spam[\"direction\"] = \"spam\"\n",
"top_ham = coefficients.nsmallest(20).reset_index()\n",
"top_ham.columns = [\"term\", \"coefficient\"]\n",
"top_ham[\"direction\"] = \"ham\"\n",
"top_terms = pd.concat([top_spam, top_ham], ignore_index=True)\n",
"top_terms.head(10)\n"
]
},
{
"cell_type": "markdown",
"id": "779dc4d3",
"metadata": {},
"source": [
"## 6. Challenge the detector with new messages\n",
"\n",
"The fourth message includes simple obfuscation. A word-based model may not recognize altered tokens such as `w1nner` or `FR33`, which exposes a robustness limitation and motivates an optional character n-gram experiment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c27e573",
"metadata": {},
"outputs": [],
"source": [
"custom_messages = pd.Series([\n",
" \"Can we move the project meeting to 3 pm?\",\n",
" \"Congratulations! Claim your free cash prize now\",\n",
" \"URGENT: verify your account at the link below\",\n",
" \"You are a w1nner, reply FR33 to collect your reward\",\n",
" \"Please call me when you arrive at the station\",\n",
"], name=\"message\")\n",
"\n",
"custom_probabilities = final_pipeline.predict_proba(custom_messages)[:, 1]\n",
"custom_results = pd.DataFrame({\n",
" \"message\": custom_messages,\n",
" \"spam_probability\": custom_probabilities,\n",
" \"predicted_label\": np.where(custom_probabilities >= DECISION_THRESHOLD, \"spam\", \"ham\"),\n",
" \"action\": pd.cut(\n",
" custom_probabilities,\n",
" bins=[-np.inf, 0.30, 0.70, np.inf],\n",
" labels=[\"allow\", \"analyst_review\", \"quarantine\"],\n",
" right=False,\n",
" ).astype(\"string\"),\n",
"})\n",
"custom_results\n"
]
},
{
"cell_type": "markdown",
"id": "a27debbf",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2d3eb2ed",
"metadata": {},
"outputs": [],
"source": [
"pd.Series(metrics, name=\"value\").to_csv(OUTPUT_DIR / \"capstone_metrics.csv\", header=True)\n",
"prediction_table.to_csv(OUTPUT_DIR / \"capstone_predictions.csv\", index=False)\n",
"top_terms.to_csv(OUTPUT_DIR / \"capstone_top_terms.csv\", index=False)\n",
"\n",
"fig, ax = plt.subplots(figsize=(5, 4))\n",
"ConfusionMatrixDisplay(\n",
" confusion_matrix=cm,\n",
" display_labels=[\"ham\", \"spam\"],\n",
").plot(ax=ax, cmap=\"Blues\", colorbar=False)\n",
"ax.set_title(\"SMS Spam Detector — Test Set\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"capstone_confusion_matrix.png\", dpi=160, bbox_inches=\"tight\")\n",
"plt.close(fig)\n",
"\n",
"summary = f'''SMS Spam Detector — Analyst Summary\n",
"===================================\n",
"Source dataset: UCI SMS Spam Collection\n",
"Clean rows: {len(clean_df)}\n",
"Training rows: {len(X_train)}\n",
"Test rows: {len(X_test)}\n",
"Overall spam rate: {y.mean():.2%}\n",
"Decision threshold: {DECISION_THRESHOLD:.2f}\n",
"\n",
"Test performance\n",
"----------------\n",
"Accuracy: {metrics['accuracy']:.3f}\n",
"Spam precision: {metrics['spam_precision']:.3f}\n",
"Spam recall: {metrics['spam_recall']:.3f}\n",
"Spam F1: {metrics['spam_f1']:.3f}\n",
"False positives: {fp}\n",
"False negatives: {fn}\n",
"\n",
"Operational interpretation\n",
"--------------------------\n",
"Messages below 0.30 are allowed, scores from 0.30 to below 0.70 are sent\n",
"for analyst review, and scores of 0.70 or more are quarantined.\n",
"\n",
"Limitation\n",
"----------\n",
"This classroom model uses SMS body text only. It does not inspect email headers,\n",
"sender domains, attachments, destination URLs, or changing attacker behavior.\n",
"'''\n",
"summary = textwrap.dedent(summary).strip() + \"\\n\"\n",
"(OUTPUT_DIR / \"capstone_summary.txt\").write_text(summary, encoding=\"utf-8\")\n",
"\n",
"print(summary)\n"
]
},
{
"cell_type": "markdown",
"id": "4ad1b775",
"metadata": {},
"source": [
"## Self-check\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a650060",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_predictions.csv\",\n",
" \"capstone_top_terms.csv\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_summary.txt\",\n",
"}\n",
"\n",
"assert set(clean_df[\"target\"].unique()) == {0, 1}\n",
"assert not clean_df.duplicated(subset=[\"label\", \"message\"]).any()\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.02\n",
"assert list(final_pipeline.named_steps) == [\"tfidf\", \"model\"]\n",
"assert len(test_predictions) == len(y_test)\n",
"assert cm.shape == (2, 2)\n",
"assert (tn + fp + fn + tp) == len(y_test)\n",
"assert len(prediction_table) == len(y_test)\n",
"assert set(prediction_table[\"action\"].unique()).issubset({\"allow\", \"analyst_review\", \"quarantine\"})\n",
"assert len(top_terms) == 40\n",
"assert len(custom_results) == 5\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"Limitation\" in summary\n",
"print(\"✓ Lab 6 completed\")\n"
]
},
{
"cell_type": "markdown",
"id": "f680874c",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **What do the false positives and false negatives mean to a security team?** \n",
" False positives consume review effort or block legitimate communication. False negatives allow spam through and may expose users to fraud or malicious links. The acceptable balance depends on organizational cost and risk.\n",
"2. **Why should the custom obfuscated message be included in the capstone?** \n",
" It checks whether the learned word vocabulary is brittle when attackers alter spelling. A failure is useful evidence for considering character n-grams or additional signals later.\n",
"3. **What evidence is still missing before this classroom model could support real email security decisions?** \n",
" The model needs representative current email data, sender and header features, URL and attachment analysis, temporal validation, calibrated policy costs, adversarial testing, drift monitoring, privacy review, and human oversight.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-4/Lab-4-Solution/data/SMSSpamCollection
+5574
View File
File diff suppressed because it is too large Load Diff
Lab-4/Lab-4-Solution/data/sms+spam+collection.zip
BIN
View File
Binary file not shown.
Lab-5/Lab-5-Practical.zip
BIN
View File
Binary file not shown.
Lab-5/Lab-5-Practical/01_Malware_Dataset_Audit_Lab.ipynb
+222
View File
@@ -0,0 +1,222 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 1: Auditing a Malware Feature Dataset\n",
"\n",
"**Topics:** provenance, schema, imbalance, missing values, leakage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load a prepared table of static and dynamic malware features safely;\n",
"- distinguish identifiers, provenance fields, features, and the binary label;\n",
"- audit class imbalance, source composition, missing values, infinite values, and duplicates;\n",
"- identify columns that must not enter a predictive model;\n",
"- document dataset limitations before modeling.\n",
"\n",
"> **Safety boundary:** This practical uses numeric feature tables only. Do not download, open, or execute malware samples."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load the prepared table\n",
"\n",
"The preparation utility adds two non-feature columns: `source` records provenance and `label` uses `0 = benign`, `1 = malware`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data/malware_features.csv into df.\n",
"df = None\n",
"\n",
"# TODO 2: verify that source and label exist, then create feature_cols.\n",
"required_columns = {\"source\", \"label\"}\n",
"feature_cols = None\n",
"\n",
"print(\"Dataset shape:\", None)\n",
"print(\"Number of candidate features:\", None)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit labels and provenance\n",
"\n",
"`source` is useful for detecting collection bias, but it must never be used as a predictor. In this dataset, collection source and class are closely related."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-007",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: calculate class counts and class proportions.\n",
"class_counts = None\n",
"class_proportions = None\n",
"\n",
"# TODO 4: create a source-by-label cross-tabulation with margins.\n",
"source_by_label = None\n",
"\n",
"display(pd.DataFrame({\"count\": class_counts, \"proportion\": class_proportions}))\n",
"display(source_by_label)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit data quality and leakage risks\n",
"\n",
"Missing values will be imputed inside a training pipeline in later labs. Imputing before the split would allow test-set statistics to influence training."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-009",
"metadata": {},
"outputs": [],
"source": [
"numeric_features = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"\n",
"# TODO 5: count infinite numeric values and exact duplicate rows.\n",
"infinite_count = None\n",
"duplicate_count = None\n",
"\n",
"# TODO 6: calculate per-feature missing rates and find constant features.\n",
"missing_summary = None\n",
"constant_features = None\n",
"\n",
"# TODO 7: find feature names containing possible label, source, filename, or hash tokens.\n",
"suspicious_tokens = (\"label\", \"class\", \"target\", \"source\", \"filename\", \"hash\", \"md5\", \"sha\")\n",
"suspicious_columns = None\n",
"\n",
"print(\"Exact duplicate rows:\", duplicate_count)\n",
"print(\"Infinite numeric values:\", infinite_count)\n",
"print(\"Constant candidate features:\", len(constant_features))\n",
"print(\"Suspicious feature names:\", suspicious_columns[:20])\n",
"display(missing_summary.head(10).to_frame())"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## 4. Write the dataset risk statement\n",
"\n",
"Record three facts before modeling:\n",
"\n",
"1. The positive class is the majority class.\n",
"2. Benign and malware rows come from different source collections.\n",
"3. A random split can measure interpolation within this historical table, but not performance on future malware campaigns."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert {\"source\", \"label\"}.issubset(df.columns)\n",
"assert len(feature_cols) > 0\n",
"assert set(df[\"label\"].dropna().unique()) == {0, 1}\n",
"assert set(feature_cols).isdisjoint({\"source\", \"label\"})\n",
"assert class_counts.loc[1] > class_counts.loc[0]\n",
"assert len(missing_summary) == len(feature_cols)\n",
"assert infinite_count == 0\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must `source` be excluded from the model?\n",
"2. Why should missing-value imputation happen after the train-test split?\n",
"3. Why is a random split not sufficient evidence for production deployment?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/02_Malware_Feature_Exploration_Lab.ipynb
+218
View File
@@ -0,0 +1,218 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
"\n",
" \n",
"**Topics:** feature families, distributions, effect size, responsible EDA\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- summarize numeric malware features by class;\n",
"- identify interpretable feature-name families;\n",
"- compare benign and malware distributions with robust statistics;\n",
"- distinguish exploratory association from causal evidence;\n",
"- avoid using the held-out test set for model-driven feature selection."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the prepared dataset and define X, y, and feature_cols.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"print(\"Rows:\", None, \"Features:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect feature-name families\n",
"\n",
"Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"family_tokens = {\n",
" \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
" \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
" \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
" \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
"}\n",
"\n",
"# TODO 2: collect up to ten feature-name examples for each family.\n",
"family_examples = None\n",
"family_examples"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build a robust class-comparison profile\n",
"\n",
"Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"benign = X.loc[y == 0]\n",
"malware = X.loc[y == 1]\n",
"\n",
"# TODO 3: calculate the IQR for every feature and replace zero IQR values with NaN.\n",
"iqr = None\n",
"\n",
"# TODO 4: build feature_profile with missing rate, unique count, and class medians.\n",
"feature_profile = None\n",
"\n",
"# TODO 5: calculate the absolute median gap divided by IQR, then select the top eight names.\n",
"top_features = None\n",
"display(feature_profile.loc[top_features])"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Visualize class medians for the strongest exploratory differences"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: create a horizontal grouped bar chart of benign and malware medians.\n",
"median_plot = None\n",
"ax = None\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert set(X.columns) == set(feature_cols)\n",
"assert set(y.unique()) == {0, 1}\n",
"assert len(feature_profile) == len(feature_cols)\n",
"assert len(top_features) == 8\n",
"assert set(top_features).issubset(feature_cols)\n",
"assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why are medians and IQR useful for security features?\n",
"2. Can a large standardized median gap prove that a feature causes malware behavior?\n",
"3. Why should this full-dataset ranking not be used directly to optimize the final model?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/03_Decision_Tree_Baseline_Lab.ipynb
+203
View File
@@ -0,0 +1,203 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 3: Building a Decision Tree Baseline\n",
"\n",
"\n",
"**Topics:** stratified split, dummy baseline, bounded tree, leakage-safe pipeline\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- create a reproducible stratified train-test split;\n",
"- fit missing-value imputation on training data only;\n",
"- compare a bounded Decision Tree with a dummy baseline;\n",
"- evaluate malware precision, recall, F1, false positives, and false negatives;\n",
"- explain why trees do not require feature scaling."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the data, exclude source and label from X, and create integer y.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"\n",
"# TODO 2: make an 80/20 stratified split with random_state=42.\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"print(\"Train shape:\", None, \"Test shape:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Establish a non-learning baseline\n",
"\n",
"Because malware is the majority class, a most-frequent classifier can achieve high accuracy while providing little operational value."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: fit a most-frequent DummyClassifier and calculate security metrics.\n",
"dummy = None\n",
"dummy_predictions = None\n",
"dummy_metrics = None\n",
"pd.Series(dummy_metrics, name=\"dummy\")"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Train a bounded Decision Tree\n",
"\n",
"The imputer is inside the pipeline, so its medians are learned from training rows only. A shallow tree is intentionally chosen for readability and to reduce overfitting."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: create a pipeline with median imputation and a Decision Tree.\n",
"# Use max_depth=4, min_samples_leaf=10, and random_state=42.\n",
"tree_pipeline = None\n",
"\n",
"# TODO 5: fit the pipeline, predict the test set, and calculate security metrics.\n",
"tree_predictions = None\n",
"tree_metrics = None\n",
"\n",
"comparison = None\n",
"display(comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"assert X_train.shape[0] + X_test.shape[0] == len(df)\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.01\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 4\n",
"assert set(comparison.index) == {\"Dummy\", \"Decision Tree\"}\n",
"assert comparison.loc[\"Decision Tree\", \"malware_f1\"] >= comparison.loc[\"Dummy\", \"malware_f1\"]\n",
"print(\"✓ Lab 3 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why can the dummy model have deceptively high accuracy?\n",
"2. Why does a Decision Tree usually not require feature scaling?\n",
"3. What is the security and interpretability benefit of limiting tree depth?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/04_Security_Metrics_Error_Costs_Lab.ipynb
+242
View File
@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 4: Security Metrics and Error Costs\n",
"\n",
" \n",
"**Topics:** false negatives, class weights, validation threshold, test discipline\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- quantify false-negative and false-positive costs;\n",
"- compare class-weight policies on validation data;\n",
"- select a probability threshold without inspecting the test set;\n",
"- evaluate the chosen policy once on held-out test data;\n",
"- describe the operational trade-off between missed malware and analyst workload."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load X and y, create a 20% stratified test set, then split the\n",
"# remaining data into 75% train and 25% validation using random_state=42.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train_full, X_test, y_train_full, y_test = None, None, None, None\n",
"X_train, X_valid, y_train, y_valid = None, None, None, None\n",
"print(\"Train/validation/test:\", None, None, None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare class-weight policies on validation data\n",
"\n",
"Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"weight_policies = {\n",
" \"unweighted\": None,\n",
" \"malware_x2\": {0: 1, 1: 2},\n",
" \"malware_x4\": {0: 1, 1: 4},\n",
"}\n",
"\n",
"# TODO 2: fit one depth-4 tree pipeline per weight policy and collect\n",
"# validation security_metrics in validation_results.\n",
"validation_results = None\n",
"fitted_candidates = None\n",
"display(validation_results)\n",
"\n",
"# TODO 3: select highest recall, breaking ties by lower FPR then higher F1.\n",
"selected_policy = None\n",
"selected_model = None\n",
"print(\"Selected class-weight policy:\", selected_policy)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a validation threshold\n",
"\n",
"Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: obtain validation malware probabilities and evaluate thresholds\n",
"# from 0.20 through 0.80 in steps of 0.05.\n",
"validation_probabilities = None\n",
"threshold_results = None\n",
"\n",
"# TODO 5: among thresholds with recall >= 0.98, select the lowest FPR.\n",
"# If none qualify, use a threshold with maximum recall.\n",
"selected_threshold = None\n",
"display(threshold_results)\n",
"print(\"Selected threshold:\", selected_threshold)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Refit on all training data and evaluate once on the test set\n",
"\n",
"The test set is used only after all policy decisions are fixed."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: refit the selected policy on X_train_full/y_train_full.\n",
"final_model = None\n",
"\n",
"# TODO 7: compare threshold 0.50 and selected_threshold on the test set.\n",
"test_probabilities = None\n",
"default_predictions = None\n",
"tuned_predictions = None\n",
"test_results = None\n",
"display(test_results)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(validation_results.index) == set(weight_policies)\n",
"assert 0.20 <= selected_threshold <= 0.80\n",
"assert len(test_probabilities) == len(y_test)\n",
"assert set(tuned_predictions).issubset({0, 1})\n",
"assert test_results.shape[0] == 2\n",
"assert final_model.named_steps[\"tree\"].get_depth() <= 4\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why must model and threshold choices use validation data rather than test data?\n",
"2. What operational harm can false negatives and false positives cause?\n",
"3. Why should the selected threshold be revalidated after deployment conditions change?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/05_Tree_Interpretation_Overfitting_Lab.ipynb
+235
View File
@@ -0,0 +1,235 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting\n",
"\n",
"\n",
"**Topics:** tree rules, complexity, feature importance, train-test gap\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare bounded and unrestricted trees;\n",
"- quantify depth, number of leaves, and train-test performance gaps;\n",
"- visualize an explainable tree;\n",
"- export root-to-leaf logic as text rules;\n",
"- interpret feature importance cautiously."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.tree import export_text, plot_tree\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, define X/y, and make an 80/20 stratified split.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train, X_test, y_train, y_test = None, None, None, None\n",
"\n",
"# TODO 2: implement make_tree and fit a bounded tree (depth 4, leaf 10)\n",
"# and an unrestricted tree (no maximum depth, leaf 1).\n",
"def make_tree(max_depth, min_samples_leaf):\n",
" pass\n",
"\n",
"bounded_tree = None\n",
"unrestricted_tree = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare model complexity and generalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create complexity_comparison with depth, leaves, train/test F1,\n",
"# F1 gap, test recall, and test false-positive rate for both trees.\n",
"complexity_comparison = None\n",
"display(complexity_comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Visualize and export the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: obtain transformed feature names, plot the bounded tree, and\n",
"# export its rules as text with max_depth=4.\n",
"imputer = None\n",
"estimator = None\n",
"transformed_names = None\n",
"rule_text = None\n",
"print(rule_text)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Inspect impurity-based feature importance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: create a descending importance Series using transformed names,\n",
"# display the top 12, and plot a horizontal bar chart.\n",
"importance = None\n",
"top_importance = None\n",
"display(top_importance)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert bounded_tree.named_steps[\"tree\"].get_depth() <= 4\n",
"assert unrestricted_tree.named_steps[\"tree\"].get_depth() >= bounded_tree.named_steps[\"tree\"].get_depth()\n",
"assert complexity_comparison.loc[\"unrestricted\", \"leaves\"] >= complexity_comparison.loc[\"bounded\", \"leaves\"]\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Which values in the comparison table indicate overfitting?\n",
"2. Translate two root-to-malware paths into `IF ... AND ... THEN malware` rules.\n",
"3. Why must feature importance not be interpreted as causal evidence?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/06_Capstone_Malware_Detection_Lab.ipynb
+275
View File
@@ -0,0 +1,275 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow\n",
"\n",
"\n",
"**Topics:** end-to-end workflow, model comparison, error evidence, analyst recommendation\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/capstone_metrics.csv`\n",
"- `output/capstone_errors.csv`\n",
"- `output/capstone_tree_rules.txt`\n",
"- `output/capstone_confusion_matrix.png`\n",
"- `output/capstone_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.tree import export_text\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the dataset, exclude source/label from X, and create an\n",
"# 80/20 stratified split that also retains source_test for error auditing.\n",
"df = None\n",
"feature_cols = None\n",
"X = None\n",
"y = None\n",
"X_train, X_test, y_train, y_test, source_train, source_test = (None,) * 6\n",
"print(\"Train/test rows:\", None, None)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Build and compare candidate models"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: define Dummy, ExplainableTree, DeepTree, and RandomForest.\n",
"# Keep ExplainableTree at max_depth=4 and min_samples_leaf=10.\n",
"models = None\n",
"\n",
"# TODO 3: fit every model, save predictions/probabilities, and build a\n",
"# metrics table with security_metrics.\n",
"metric_rows = None\n",
"predictions = None\n",
"probabilities = None\n",
"metrics_table = None\n",
"display(metrics_table)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a model with an explicit decision rule\n",
"\n",
"Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: implement the documented selection rule.\n",
"eligible_metrics = None\n",
"best_recall = None\n",
"best_fpr = None\n",
"explainable_ok = None\n",
"selected_name = None\n",
"selected_model = None\n",
"selected_predictions = None\n",
"print(\"Selected model:\", selected_name)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Create error evidence and a confusion matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: create error_table with source, true label, prediction,\n",
"# malware probability, and error_type. Keep mistakes in errors_only.\n",
"error_table = None\n",
"errors_only = None\n",
"display(errors_only[\"error_type\"].value_counts())\n",
"\n",
"# TODO 6: plot and save the selected model's confusion matrix."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export rules and write the analyst recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: export the ExplainableTree rules even if another model was selected.\n",
"tree_rules = None\n",
"\n",
"# TODO 8: write an English analyst summary covering the recommendation,\n",
"# metrics, FP/FN costs, data limitations, and deployment safeguards.\n",
"analyst_summary = None\n",
"\n",
"# TODO 9: save the metrics, errors, rules, and analyst summary.\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_errors.csv\",\n",
" \"capstone_tree_rules.txt\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_analyst_summary.md\",\n",
"}\n",
"assert set(models) == {\"Dummy\", \"ExplainableTree\", \"DeepTree\", \"RandomForest\"}\n",
"assert selected_name in set(models) - {\"Dummy\"}\n",
"assert len(error_table) == len(y_test)\n",
"assert set(errors_only[\"error_type\"].unique()).issubset({\"false_negative\", \"false_positive\"})\n",
"assert isinstance(tree_rules, str) and \"class:\" in tree_rules\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom.\n",
"2. Confirm that all five output files exist.\n",
"3. Verify that the recommendation is supported by the metrics table.\n",
"4. Explain at least one false negative and one false positive operationally.\n",
"5. State why this random-split result is not a production guarantee."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Practical/data/README.md
+79
View File
@@ -0,0 +1,79 @@
# Dataset setup — UCI Malware Static and Dynamic Features
All six notebooks use pre-extracted tabular features from the official UCI dataset **Malware Static and Dynamic Features VxHeaven and Virus Total**.
## Download
1. Open the [UCI dataset page](https://archive.ics.uci.edu/dataset/541/malware%2Bstatic%2Band%2Bdynamic%2Bfeatures%2Bvxheaven%2Band%2Bvirus%2Btotal).
2. Select **Download** and extract the archive.
3. Create `data/raw/` if it does not already exist.
4. Copy these three files into `data/raw/`:
```text
staDynBenignLab.csv
staDynVxHeaven2698Lab.csv
staDynVt2955Lab.csv
```
The expected layout is:
```text
Practical-5/
└── data/
├── prepare_dataset.py
└── raw/
├── staDynBenignLab.csv
├── staDynVxHeaven2698Lab.csv
└── staDynVt2955Lab.csv
```
## Prepare the classroom table
From the `Practical-5` directory, run:
```bash
python data/prepare_dataset.py
```
The utility:
- finds the three required files under `data/raw/`;
- normalizes column names and adds deterministic occurrence suffixes when
case- or punctuation-distinct UCI fields collapse to the same snake-case name;
- keeps columns shared by all three sources;
- removes obvious identifiers and pre-existing target columns;
- converts usable columns to numeric values;
- adds `source` and `label` (`0 = benign`, `1 = malware`);
- retains missing values so imputation can be fitted on training data inside the notebooks;
- writes `data/malware_features.csv` and `data/feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
path = Path("data/malware_features.csv")
df = pd.read_csv(path)
print(df.shape)
print(df["label"].value_counts().sort_index())
print(df["source"].value_counts())
```
The original release contains approximately 6,248 rows: 595 benign records and 5,653 malware records. The exact number of retained feature columns can vary if the source archive changes column formatting.
## Provenance and license
- Dataset: Malware Static and Dynamic Features VxHeaven and Virus Total
- Source: UCI Machine Learning Repository
- DOI: [10.24432/C58K6H](https://doi.org/10.24432/C58K6H)
- License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
Retain the citation and license when redistributing derived classroom tables.
## Important limitations
- The class distribution is strongly imbalanced.
- Benign and malware samples originate from different collections, so source-specific artifacts may inflate random-split performance.
- The features represent a historical dataset and should not be treated as a current production benchmark.
- No raw malware is required or permitted for this practical.
Lab-5/Lab-5-Practical/data/feature_manifest.csv
+245
View File
@@ -0,0 +1,245 @@
feature,non_missing_ratio,unique_values,feature_family
add,1.0,2809,unclassified
addressofentrypoint,1.0,3156,unclassified
align,1.0,11,unclassified
arg,1.0,5,unclassified
asc,1.0,3,unclassified
baseofcode,1.0,145,unclassified
bool,1.0,2,unclassified
bt,1.0,114,unclassified
byte,1.0,17,unclassified
call,1.0,2131,unclassified
cexit,1.0,2,unclassified
char,1.0,4,unclassified
characteristics,1.0,52,unclassified
checksum,1.0,2100,unclassified
cld,1.0,685,unclassified
cli,1.0,625,unclassified
closehandle,1.0,2,unclassified
cmc,1.0,635,unclassified
cmp,1.0,1954,unclassified
compile_date,1.0,3912,unclassified
cookie,1.0,2,unclassified
copyfilea,1.0,2,unclassified
count_dll_loaded,1.0,60,unclassified
count_file_copied,1.0,47,unclassified
count_file_deleted,1.0,19,unclassified
count_file_exists,1.0,75,unclassified
count_file_opened,1.0,76,unclassified
count_file_read,1.0,29,unclassified
count_file_written,1.0,24,unclassified
count_mutex,1.0,18,unclassified
count_regkey_deleted,1.0,12,unclassified
count_regkey_written,1.0,45,unclassified
createdirectorya,1.0,2,unclassified
createfilea,1.0,2,unclassified
createmutexa,1.0,2,unclassified
createprocessa,1.0,2,unclassified
createthread,1.0,2,unclassified
createtoolhelp32snapshot,1.0,2,unclassified
cs,1.0,529,unclassified
daa,1.0,618,unclassified
data,1.0,4,unclassified
data__2,1.0,2,unclassified
db,1.0,1827,unclassified
db_1,1.0,1147,unclassified
dd,1.0,3111,unclassified
dd_1,1.0,946,unclassified
dec,1.0,1627,unclassified
deletefilea,1.0,2,unclassified
dll,1.0,225,unclassified
dll_characteristics,1.0,27,unclassified
ds,1.0,526,unclassified
dw,1.0,63,unclassified
enablewindow,1.0,2,unclassified
ent_max,1.0,5674,unclassified
ent_mean,1.0,6243,unclassified
ent_median,1.0,6087,unclassified
ent_min,1.0,5699,unclassified
ent_var,1.0,6114,unclassified
ent_whole_file,1.0,6248,unclassified
entry,1.0,5,unclassified
es,1.0,500,unclassified
exit,1.0,33,unclassified
exitprocess,1.0,3,unclassified
exitwindowsex,1.0,2,unclassified
faddp,1.0,131,unclassified
fchs,1.0,55,unclassified
fclose,1.0,2,unclassified
fdiv,1.0,175,unclassified
fdivp,1.0,130,unclassified
fdivr,1.0,192,unclassified
fild,1.0,276,unclassified
file_alignment,1.0,5,unclassified
files_operations,1.0,159,unclassified
filesize,1.0,5252,unclassified
findclose,1.0,2,unclassified
findfirstfilea,1.0,2,unclassified
findnextfilea,1.0,2,unclassified
findwindowa,1.0,2,unclassified
fistp,1.0,202,unclassified
fld,1.0,144,unclassified
fopen,1.0,3,unclassified
free,1.0,169,unclassified
fs,1.0,683,unclassified
fstcw,1.0,19,unclassified
fwrite,1.0,3,unclassified
fxch,1.0,148,unclassified
getcommandlinea,1.0,2,unclassified
getfileattributesa,1.0,2,unclassified
getforegroundwindow,1.0,2,unclassified
getlasterror,1.0,4,unclassified
getmainargs,1.0,2,unclassified
getmodulefilenamea,1.0,3,unclassified
getmodulehandlea,1.0,3,unclassified
getstartupinfoa,1.0,2,unclassified
getsystemdirectorya,1.0,2,unclassified
getsystemmetrics,1.0,2,unclassified
gettickcount,1.0,2,unclassified
gs,1.0,796,unclassified
handle,1.0,10,unclassified
image_base,1.0,44,unclassified
imul,1.0,1113,unclassified
in,1.0,1016,unclassified
inc,1.0,1941,unclassified
initterm,1.0,2,unclassified
int,1.0,1500,unclassified
int_1,1.0,575,unclassified
interlockedcompareexchange,1.0,2,unclassified
interlockeddecrement,1.0,2,unclassified
interlockedincrement,1.0,2,unclassified
jb,1.0,1089,unclassified
je,1.0,1732,unclassified
jg,1.0,765,unclassified
jge,1.0,748,unclassified
jl,1.0,825,unclassified
jmp,1.0,1707,unclassified
jno,1.0,650,unclassified
jo,1.0,819,unclassified
lea,1.0,1673,unclassified
loader_flags,1.0,4,unclassified
loc,1.0,688,unclassified
long,1.0,10,unclassified
loope,1.0,580,unclassified
lstrcata,1.0,2,unclassified
lstrcpya,1.0,2,unclassified
lstrlena,1.0,2,unclassified
machine,1.0,2,unclassified
magic,1.0,2,unclassified
major_image_version,1.0,39,unclassified
major_linker_version,1.0,30,unclassified
major_operating_system_version,1.0,7,unclassified
major_subsystem_version,1.0,5,unclassified
malloc,1.0,5,unclassified
max_min,1.0,5517,unclassified
memcpy,1.0,5,unclassified
memmove,1.0,3,unclassified
memset,1.0,5,unclassified
messageboxa,1.0,2,unclassified
minor_image_version,1.0,44,unclassified
minor_linker_version,1.0,47,unclassified
minor_operating_system_version,1.0,10,unclassified
minor_subsystem_version,1.0,6,unclassified
mov,1.0,3156,unclassified
mul,1.0,207,unclassified
neg,1.0,301,unclassified
nop,1.0,1037,unclassified
not,1.0,265,unclassified
number_of_iat_entires,1.0,467,unclassified
number_of_iat_entires_1,1.0,467,unclassified
number_of_imports_1,1.0,40,unclassified
number_of_rva_and_sizes,1.0,8,unclassified
number_of_rva_and_sizes_1,1.0,8,unclassified
number_of_sections_1,1.0,29,unclassified
number_of_symbols,1.0,50,unclassified
off,1.0,4,unclassified
or,1.0,1497,unclassified
out,1.0,1013,unclassified
p_fmode,1.0,2,unclassified
pointer_to_symbol_table,1.0,42,unclassified
pop,1.0,2306,unclassified
popf,1.0,585,unclassified
proc,1.0,2,unclassified
process32first,1.0,2,unclassified
process32next,1.0,2,unclassified
ptr,1.0,16,unclassified
push,1.0,3002,unclassified
pushf,1.0,642,unclassified
raiseexception,1.0,3,unclassified
rand,1.0,6,unclassified
rcl,1.0,314,unclassified
rcr,1.0,261,unclassified
rdata,1.0,2,unclassified
rdtsc,1.0,50,unclassified
regclosekey,1.0,3,unclassified
regopenkeya,1.0,2,unclassified
regopenkeyexa,1.0,2,unclassified
regsetvalueexa,1.0,3,unclassified
rep,1.0,263,unclassified
ret,1.0,1458,unclassified
rol,1.0,348,unclassified
ror,1.0,332,unclassified
sar,1.0,466,unclassified
sbb,1.0,1230,unclassified
scas,1.0,777,unclassified
section_alignment,1.0,5,unclassified
sendmessagea,1.0,2,unclassified
set_app_type,1.0,3,unclassified
setb,1.0,54,unclassified
setcursorpos,1.0,2,unclassified
setfileattributesa,1.0,3,unclassified
setlasterror,1.0,2,unclassified
setle,1.0,50,unclassified
setusermatherr,1.0,2,unclassified
setwindowtexta,1.0,2,unclassified
shellexecutea,1.0,2,unclassified
shl,1.0,493,unclassified
shld,1.0,79,unclassified
showwindow,1.0,2,unclassified
shr,1.0,499,unclassified
sidt,1.0,2,unclassified
size_code,1.0,930,unclassified
size_image,1.0,829,unclassified
size_init_data,1.0,948,unclassified
size_of_headers,1.0,30,unclassified
size_of_heap_commit,1.0,10,unclassified
size_of_heap_reserve,1.0,26,unclassified
size_of_optional_header,1.0,2,unclassified
size_of_stack_commit,1.0,35,unclassified
size_of_stack_reserve,1.0,34,unclassified
size_uninit_data,1.0,164,unclassified
sizeofheaders,1.0,30,unclassified
sizeofheaders_1,1.0,30,unclassified
sleep,1.0,4,unclassified
sprintf,1.0,2,unclassified
ss,1.0,577,unclassified
stc,1.0,634,unclassified
std,1.0,658,unclassified
sti,1.0,669,unclassified
stos,1.0,790,unclassified
strcat,1.0,4,unclassified
strchr,1.0,2,unclassified
strcmp,1.0,4,unclassified
strcpy,1.0,4,unclassified
strlen,1.0,6,unclassified
strncmp,1.0,2,unclassified
strstr,1.0,3,unclassified
sub,1.0,1631,unclassified
subsystem,1.0,3,unclassified
test,1.0,1685,unclassified
text,1.0,5,unclassified
time,1.0,5,unclassified
tlsalloc,1.0,2,unclassified
tlsfree,1.0,2,unclassified
tlsgetvalue,1.0,2,unclassified
tlssetvalue,1.0,2,unclassified
uint,1.0,3,unclassified
unhandledexceptionfilter,1.0,2,unclassified
waitforsingleobject,1.0,2,unclassified
winexec,1.0,2,unclassified
winmain,1.0,6,unclassified
word,1.0,9,unclassified
word__2,1.0,3,unclassified
writefile,1.0,2,unclassified
xchg,1.0,1217,unclassified
xor,1.0,1887,unclassified
1 feature non_missing_ratio unique_values feature_family
2 add 1.0 2809 unclassified
3 addressofentrypoint 1.0 3156 unclassified
4 align 1.0 11 unclassified
5 arg 1.0 5 unclassified
6 asc 1.0 3 unclassified
7 baseofcode 1.0 145 unclassified
8 bool 1.0 2 unclassified
9 bt 1.0 114 unclassified
10 byte 1.0 17 unclassified
11 call 1.0 2131 unclassified
12 cexit 1.0 2 unclassified
13 char 1.0 4 unclassified
14 characteristics 1.0 52 unclassified
15 checksum 1.0 2100 unclassified
16 cld 1.0 685 unclassified
17 cli 1.0 625 unclassified
18 closehandle 1.0 2 unclassified
19 cmc 1.0 635 unclassified
20 cmp 1.0 1954 unclassified
21 compile_date 1.0 3912 unclassified
22 cookie 1.0 2 unclassified
23 copyfilea 1.0 2 unclassified
24 count_dll_loaded 1.0 60 unclassified
25 count_file_copied 1.0 47 unclassified
26 count_file_deleted 1.0 19 unclassified
27 count_file_exists 1.0 75 unclassified
28 count_file_opened 1.0 76 unclassified
29 count_file_read 1.0 29 unclassified
30 count_file_written 1.0 24 unclassified
31 count_mutex 1.0 18 unclassified
32 count_regkey_deleted 1.0 12 unclassified
33 count_regkey_written 1.0 45 unclassified
34 createdirectorya 1.0 2 unclassified
35 createfilea 1.0 2 unclassified
36 createmutexa 1.0 2 unclassified
37 createprocessa 1.0 2 unclassified
38 createthread 1.0 2 unclassified
39 createtoolhelp32snapshot 1.0 2 unclassified
40 cs 1.0 529 unclassified
41 daa 1.0 618 unclassified
42 data 1.0 4 unclassified
43 data__2 1.0 2 unclassified
44 db 1.0 1827 unclassified
45 db_1 1.0 1147 unclassified
46 dd 1.0 3111 unclassified
47 dd_1 1.0 946 unclassified
48 dec 1.0 1627 unclassified
49 deletefilea 1.0 2 unclassified
50 dll 1.0 225 unclassified
51 dll_characteristics 1.0 27 unclassified
52 ds 1.0 526 unclassified
53 dw 1.0 63 unclassified
54 enablewindow 1.0 2 unclassified
55 ent_max 1.0 5674 unclassified
56 ent_mean 1.0 6243 unclassified
57 ent_median 1.0 6087 unclassified
58 ent_min 1.0 5699 unclassified
59 ent_var 1.0 6114 unclassified
60 ent_whole_file 1.0 6248 unclassified
61 entry 1.0 5 unclassified
62 es 1.0 500 unclassified
63 exit 1.0 33 unclassified
64 exitprocess 1.0 3 unclassified
65 exitwindowsex 1.0 2 unclassified
66 faddp 1.0 131 unclassified
67 fchs 1.0 55 unclassified
68 fclose 1.0 2 unclassified
69 fdiv 1.0 175 unclassified
70 fdivp 1.0 130 unclassified
71 fdivr 1.0 192 unclassified
72 fild 1.0 276 unclassified
73 file_alignment 1.0 5 unclassified
74 files_operations 1.0 159 unclassified
75 filesize 1.0 5252 unclassified
76 findclose 1.0 2 unclassified
77 findfirstfilea 1.0 2 unclassified
78 findnextfilea 1.0 2 unclassified
79 findwindowa 1.0 2 unclassified
80 fistp 1.0 202 unclassified
81 fld 1.0 144 unclassified
82 fopen 1.0 3 unclassified
83 free 1.0 169 unclassified
84 fs 1.0 683 unclassified
85 fstcw 1.0 19 unclassified
86 fwrite 1.0 3 unclassified
87 fxch 1.0 148 unclassified
88 getcommandlinea 1.0 2 unclassified
89 getfileattributesa 1.0 2 unclassified
90 getforegroundwindow 1.0 2 unclassified
91 getlasterror 1.0 4 unclassified
92 getmainargs 1.0 2 unclassified
93 getmodulefilenamea 1.0 3 unclassified
94 getmodulehandlea 1.0 3 unclassified
95 getstartupinfoa 1.0 2 unclassified
96 getsystemdirectorya 1.0 2 unclassified
97 getsystemmetrics 1.0 2 unclassified
98 gettickcount 1.0 2 unclassified
99 gs 1.0 796 unclassified
100 handle 1.0 10 unclassified
101 image_base 1.0 44 unclassified
102 imul 1.0 1113 unclassified
103 in 1.0 1016 unclassified
104 inc 1.0 1941 unclassified
105 initterm 1.0 2 unclassified
106 int 1.0 1500 unclassified
107 int_1 1.0 575 unclassified
108 interlockedcompareexchange 1.0 2 unclassified
109 interlockeddecrement 1.0 2 unclassified
110 interlockedincrement 1.0 2 unclassified
111 jb 1.0 1089 unclassified
112 je 1.0 1732 unclassified
113 jg 1.0 765 unclassified
114 jge 1.0 748 unclassified
115 jl 1.0 825 unclassified
116 jmp 1.0 1707 unclassified
117 jno 1.0 650 unclassified
118 jo 1.0 819 unclassified
119 lea 1.0 1673 unclassified
120 loader_flags 1.0 4 unclassified
121 loc 1.0 688 unclassified
122 long 1.0 10 unclassified
123 loope 1.0 580 unclassified
124 lstrcata 1.0 2 unclassified
125 lstrcpya 1.0 2 unclassified
126 lstrlena 1.0 2 unclassified
127 machine 1.0 2 unclassified
128 magic 1.0 2 unclassified
129 major_image_version 1.0 39 unclassified
130 major_linker_version 1.0 30 unclassified
131 major_operating_system_version 1.0 7 unclassified
132 major_subsystem_version 1.0 5 unclassified
133 malloc 1.0 5 unclassified
134 max_min 1.0 5517 unclassified
135 memcpy 1.0 5 unclassified
136 memmove 1.0 3 unclassified
137 memset 1.0 5 unclassified
138 messageboxa 1.0 2 unclassified
139 minor_image_version 1.0 44 unclassified
140 minor_linker_version 1.0 47 unclassified
141 minor_operating_system_version 1.0 10 unclassified
142 minor_subsystem_version 1.0 6 unclassified
143 mov 1.0 3156 unclassified
144 mul 1.0 207 unclassified
145 neg 1.0 301 unclassified
146 nop 1.0 1037 unclassified
147 not 1.0 265 unclassified
148 number_of_iat_entires 1.0 467 unclassified
149 number_of_iat_entires_1 1.0 467 unclassified
150 number_of_imports_1 1.0 40 unclassified
151 number_of_rva_and_sizes 1.0 8 unclassified
152 number_of_rva_and_sizes_1 1.0 8 unclassified
153 number_of_sections_1 1.0 29 unclassified
154 number_of_symbols 1.0 50 unclassified
155 off 1.0 4 unclassified
156 or 1.0 1497 unclassified
157 out 1.0 1013 unclassified
158 p_fmode 1.0 2 unclassified
159 pointer_to_symbol_table 1.0 42 unclassified
160 pop 1.0 2306 unclassified
161 popf 1.0 585 unclassified
162 proc 1.0 2 unclassified
163 process32first 1.0 2 unclassified
164 process32next 1.0 2 unclassified
165 ptr 1.0 16 unclassified
166 push 1.0 3002 unclassified
167 pushf 1.0 642 unclassified
168 raiseexception 1.0 3 unclassified
169 rand 1.0 6 unclassified
170 rcl 1.0 314 unclassified
171 rcr 1.0 261 unclassified
172 rdata 1.0 2 unclassified
173 rdtsc 1.0 50 unclassified
174 regclosekey 1.0 3 unclassified
175 regopenkeya 1.0 2 unclassified
176 regopenkeyexa 1.0 2 unclassified
177 regsetvalueexa 1.0 3 unclassified
178 rep 1.0 263 unclassified
179 ret 1.0 1458 unclassified
180 rol 1.0 348 unclassified
181 ror 1.0 332 unclassified
182 sar 1.0 466 unclassified
183 sbb 1.0 1230 unclassified
184 scas 1.0 777 unclassified
185 section_alignment 1.0 5 unclassified
186 sendmessagea 1.0 2 unclassified
187 set_app_type 1.0 3 unclassified
188 setb 1.0 54 unclassified
189 setcursorpos 1.0 2 unclassified
190 setfileattributesa 1.0 3 unclassified
191 setlasterror 1.0 2 unclassified
192 setle 1.0 50 unclassified
193 setusermatherr 1.0 2 unclassified
194 setwindowtexta 1.0 2 unclassified
195 shellexecutea 1.0 2 unclassified
196 shl 1.0 493 unclassified
197 shld 1.0 79 unclassified
198 showwindow 1.0 2 unclassified
199 shr 1.0 499 unclassified
200 sidt 1.0 2 unclassified
201 size_code 1.0 930 unclassified
202 size_image 1.0 829 unclassified
203 size_init_data 1.0 948 unclassified
204 size_of_headers 1.0 30 unclassified
205 size_of_heap_commit 1.0 10 unclassified
206 size_of_heap_reserve 1.0 26 unclassified
207 size_of_optional_header 1.0 2 unclassified
208 size_of_stack_commit 1.0 35 unclassified
209 size_of_stack_reserve 1.0 34 unclassified
210 size_uninit_data 1.0 164 unclassified
211 sizeofheaders 1.0 30 unclassified
212 sizeofheaders_1 1.0 30 unclassified
213 sleep 1.0 4 unclassified
214 sprintf 1.0 2 unclassified
215 ss 1.0 577 unclassified
216 stc 1.0 634 unclassified
217 std 1.0 658 unclassified
218 sti 1.0 669 unclassified
219 stos 1.0 790 unclassified
220 strcat 1.0 4 unclassified
221 strchr 1.0 2 unclassified
222 strcmp 1.0 4 unclassified
223 strcpy 1.0 4 unclassified
224 strlen 1.0 6 unclassified
225 strncmp 1.0 2 unclassified
226 strstr 1.0 3 unclassified
227 sub 1.0 1631 unclassified
228 subsystem 1.0 3 unclassified
229 test 1.0 1685 unclassified
230 text 1.0 5 unclassified
231 time 1.0 5 unclassified
232 tlsalloc 1.0 2 unclassified
233 tlsfree 1.0 2 unclassified
234 tlsgetvalue 1.0 2 unclassified
235 tlssetvalue 1.0 2 unclassified
236 uint 1.0 3 unclassified
237 unhandledexceptionfilter 1.0 2 unclassified
238 waitforsingleobject 1.0 2 unclassified
239 winexec 1.0 2 unclassified
240 winmain 1.0 6 unclassified
241 word 1.0 9 unclassified
242 word__2 1.0 3 unclassified
243 writefile 1.0 2 unclassified
244 xchg 1.0 1217 unclassified
245 xor 1.0 1887 unclassified
Lab-5/Lab-5-Practical/data/malware_features.csv
+6249
View File
File diff suppressed because it is too large Load Diff
Lab-5/Lab-5-Practical/data/prepare_dataset.py
+168
View File
@@ -0,0 +1,168 @@
"""Prepare a safe classroom feature table from the three UCI CSV files."""
from pathlib import Path
import re
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
OUTPUT_PATH = HERE / "malware_features.csv"
MANIFEST_PATH = HERE / "feature_manifest.csv"
FILES = {
"benign_lab": ("staDynBenignLab.csv", 0),
"vxheaven": ("staDynVxHeaven2698Lab.csv", 1),
"virustotal": ("staDynVt2955Lab.csv", 1),
}
FORBIDDEN_NAMES = {
"label", "class", "target", "malware", "is_malware", "type",
"file", "filename", "file_name", "name", "hash", "md5", "sha1",
"sha256", "source", "category",
}
def normalize_name(value: object) -> str:
"""Convert a source column name to a stable snake_case identifier."""
text = re.sub(r"[^0-9A-Za-z]+", "_", str(value).strip()).strip("_").lower()
return text or "unnamed_feature"
def normalize_columns(columns: list[object]) -> tuple[list[str], dict[str, list[str]]]:
"""Normalize names and disambiguate collisions by their source-column order.
The UCI files intentionally contain case- or punctuation-distinct fields such as
``WORD``/``word`` and ``Import``/``__IMPORT_``. A simple lowercase snake-case
conversion collapses each pair. The three source files use the same relative
order for these fields, so an occurrence suffix produces stable shared names:
``word``, ``word__2``, and so on.
"""
counts: dict[str, int] = {}
normalized: list[str] = []
originals_by_base: dict[str, list[str]] = {}
for raw_column in columns:
base = normalize_name(raw_column)
counts[base] = counts.get(base, 0) + 1
occurrence = counts[base]
unique_name = base if occurrence == 1 else f"{base}__{occurrence}"
normalized.append(unique_name)
originals_by_base.setdefault(base, []).append(str(raw_column))
collisions = {
base: originals
for base, originals in originals_by_base.items()
if len(originals) > 1
}
if len(normalized) != len(set(normalized)):
raise ValueError("Column names are still duplicated after disambiguation.")
return normalized, collisions
def find_required_file(filename: str) -> Path:
"""Find one required CSV below data/raw, allowing an extracted subfolder."""
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and check the extracted folder structure."
)
return matches[0]
def load_source(source: str, filename: str, label: int) -> pd.DataFrame:
"""Load one source file and attach provenance columns."""
frame = pd.read_csv(find_required_file(filename), low_memory=False)
normalized, collisions = normalize_columns(frame.columns.tolist())
frame.columns = normalized
# Consolidate the wide table before adding provenance columns. This avoids
# pandas fragmentation warnings on the 1,000+ column UCI source files.
frame = frame.copy()
frame.attrs["normalized_column_collisions"] = collisions
frame["source"] = source
frame["label"] = label
return frame
def validate_collision_alignment(frames: dict[str, pd.DataFrame]) -> None:
"""Ensure occurrence suffixes refer to the same raw fields in every source."""
collision_names = set().union(*(
set(frame.attrs.get("normalized_column_collisions", {}))
for frame in frames.values()
))
for base in sorted(collision_names):
variants = {
source: tuple(frame.attrs.get("normalized_column_collisions", {}).get(base, []))
for source, frame in frames.items()
}
if len(set(variants.values())) != 1:
raise ValueError(
f"Normalized collision {base!r} is not aligned across source files: {variants}"
)
if collision_names:
print("Disambiguated normalized column collisions:")
reference = next(iter(frames.values())).attrs["normalized_column_collisions"]
for base in sorted(collision_names):
generated = [base] + [f"{base}__{index}" for index in range(2, len(reference[base]) + 1)]
print(f" {reference[base]} -> {generated}")
def main() -> None:
frames = {
source: load_source(source, filename, label)
for source, (filename, label) in FILES.items()
}
validate_collision_alignment(frames)
common = set.intersection(*(set(frame.columns) for frame in frames.values()))
candidate_features = sorted(common - FORBIDDEN_NAMES - {"source", "label"})
if not candidate_features:
raise ValueError("No shared candidate feature columns were found.")
combined_parts = []
for source, frame in frames.items():
part = frame[candidate_features].copy()
part.insert(0, "source", source)
part.insert(1, "label", int(FILES[source][1]))
combined_parts.append(part)
combined = pd.concat(combined_parts, ignore_index=True)
numeric = combined[candidate_features].apply(pd.to_numeric, errors="coerce")
numeric = numeric.replace([np.inf, -np.inf], np.nan)
usable_ratio = numeric.notna().mean()
retained = usable_ratio[usable_ratio >= 0.95].index.tolist()
numeric = numeric[retained]
non_constant = numeric.nunique(dropna=True)
retained = non_constant[non_constant > 1].index.tolist()
numeric = numeric[retained]
if numeric.empty:
raise ValueError("No usable numeric features remained after validation.")
prepared = pd.concat([combined[["source", "label"]], numeric], axis=1)
prepared = prepared.sample(frac=1.0, random_state=42).reset_index(drop=True)
prepared.to_csv(OUTPUT_PATH, index=False)
manifest = pd.DataFrame({
"feature": retained,
"non_missing_ratio": numeric[retained].notna().mean().values,
"unique_values": numeric[retained].nunique(dropna=True).values,
"feature_family": "unclassified",
})
manifest.to_csv(MANIFEST_PATH, index=False)
print(f"Created {OUTPUT_PATH}")
print(f"Shape: {prepared.shape}")
print("Class counts:")
print(prepared["label"].value_counts().sort_index())
print("Source counts:")
print(prepared["source"].value_counts())
print(f"Created {MANIFEST_PATH}")
if __name__ == "__main__":
main()
Lab-5/Lab-5-Practical/data/raw/staDynBenignLab.csv
+596
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Practical/data/raw/staDynVt2955Lab.csv
+2956
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Practical/data/raw/staDynVxHeaven2698Lab.csv
+2699
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Solution.zip
BIN
View File
Binary file not shown.
Lab-5/Lab-5-Solution/01_Malware_Dataset_Audit_Solution.ipynb
+235
View File
@@ -0,0 +1,235 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 1: Auditing a Malware Feature Dataset\n",
"\n",
"\n",
"**Topics:** provenance, schema, imbalance, missing values, leakage\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- load a prepared table of static and dynamic malware features safely;\n",
"- distinguish identifiers, provenance fields, features, and the binary label;\n",
"- audit class imbalance, source composition, missing values, infinite values, and duplicates;\n",
"- identify columns that must not enter a predictive model;\n",
"- document dataset limitations before modeling.\n",
"\n",
"> **Safety boundary:** This practical uses numeric feature tables only. Do not download, open, or execute malware samples."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load the prepared table\n",
"\n",
"The preparation utility adds two non-feature columns: `source` records provenance and `label` uses `0 = benign`, `1 = malware`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"required_columns = {\"source\", \"label\"}\n",
"if not required_columns.issubset(df.columns):\n",
" raise ValueError(f\"Missing required columns: {required_columns - set(df.columns)}\")\n",
"\n",
"feature_cols = [column for column in df.columns if column not in required_columns]\n",
"print(\"Dataset shape:\", df.shape)\n",
"print(\"Number of candidate features:\", len(feature_cols))\n",
"display(df.head())"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit labels and provenance\n",
"\n",
"`source` is useful for detecting collection bias, but it must never be used as a predictor. In this dataset, collection source and class are closely related."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-007",
"metadata": {},
"outputs": [],
"source": [
"class_counts = df[\"label\"].value_counts().sort_index()\n",
"class_proportions = df[\"label\"].value_counts(normalize=True).sort_index()\n",
"source_by_label = pd.crosstab(df[\"source\"], df[\"label\"], margins=True)\n",
"\n",
"print(\"Class counts (0 = benign, 1 = malware):\")\n",
"display(pd.DataFrame({\"count\": class_counts, \"proportion\": class_proportions}))\n",
"print(\"Source by label:\")\n",
"display(source_by_label)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit data quality and leakage risks\n",
"\n",
"Missing values will be imputed inside a training pipeline in later labs. Imputing before the split would allow test-set statistics to influence training."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-009",
"metadata": {},
"outputs": [],
"source": [
"numeric_features = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"infinite_count = int(np.isinf(numeric_features.to_numpy(dtype=float)).sum())\n",
"missing_summary = numeric_features.isna().mean().sort_values(ascending=False).rename(\"missing_rate\")\n",
"duplicate_count = int(df.duplicated().sum())\n",
"constant_features = numeric_features.nunique(dropna=True).loc[lambda values: values <= 1].index.tolist()\n",
"\n",
"suspicious_tokens = (\"label\", \"class\", \"target\", \"source\", \"filename\", \"hash\", \"md5\", \"sha\")\n",
"suspicious_columns = [\n",
" column for column in feature_cols\n",
" if any(token in column.lower() for token in suspicious_tokens)\n",
"]\n",
"\n",
"print(\"Exact duplicate rows:\", duplicate_count)\n",
"print(\"Infinite numeric values:\", infinite_count)\n",
"print(\"Constant candidate features:\", len(constant_features))\n",
"print(\"Suspicious feature names:\", suspicious_columns[:20])\n",
"display(missing_summary.head(10).to_frame())"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## 4. Write the dataset risk statement\n",
"\n",
"Record three facts before modeling:\n",
"\n",
"1. The positive class is the majority class.\n",
"2. Benign and malware rows come from different source collections.\n",
"3. A random split can measure interpolation within this historical table, but not performance on future malware campaigns."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert {\"source\", \"label\"}.issubset(df.columns)\n",
"assert len(feature_cols) > 0\n",
"assert set(df[\"label\"].dropna().unique()) == {0, 1}\n",
"assert set(feature_cols).isdisjoint({\"source\", \"label\"})\n",
"assert class_counts.loc[1] > class_counts.loc[0]\n",
"assert len(missing_summary) == len(feature_cols)\n",
"assert infinite_count == 0\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why exclude `source`?** The source is strongly associated with the label. A model could learn the collection process instead of malware behavior.\n",
"2. **Why delay imputation?** The imputer must learn medians from training data only; otherwise test information leaks into training.\n",
"3. **Why is a random split limited?** Samples from the same historical collections appear on both sides of the split, so the test set does not represent future drift or a new collection process."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c3b5e17-8d80-4c4b-a68f-7d6e085b04f0",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "a20ec3d1-331a-46ab-8e18-f3cd73ff9fc3",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/02_Malware_Feature_Exploration_Solution.ipynb
+227
View File
@@ -0,0 +1,227 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 2: Exploring Malware Feature Representations\n",
"\n",
"\n",
"**Topics:** feature families, distributions, effect size, responsible EDA\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- summarize numeric malware features by class;\n",
"- identify interpretable feature-name families;\n",
"- compare benign and malware distributions with robust statistics;\n",
"- distinguish exploratory association from causal evidence;\n",
"- avoid using the held-out test set for model-driven feature selection."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"y = df[\"label\"].astype(int)\n",
"print(\"Rows:\", len(df), \"Features:\", len(feature_cols))"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect feature-name families\n",
"\n",
"Real malware datasets may contain PE-header fields, strings/imports, byte or opcode statistics, and sandbox behavior. Naming conventions are imperfect, so this step is an audit rather than a guaranteed taxonomy."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"family_tokens = {\n",
" \"pe_or_header\": (\"header\", \"section\", \"image\", \"size\", \"character\", \"linker\", \"dll\", \"entry\"),\n",
" \"api_or_import\": (\"api\", \"import\", \"call\", \"function\"),\n",
" \"byte_or_opcode\": (\"byte\", \"opcode\", \"asm\", \"hex\"),\n",
" \"dynamic_behavior\": (\"registry\", \"network\", \"process\", \"file\", \"mutex\", \"service\"),\n",
"}\n",
"\n",
"family_examples = {}\n",
"for family, tokens in family_tokens.items():\n",
" matches = [name for name in feature_cols if any(token in name.lower() for token in tokens)]\n",
" family_examples[family] = matches[:10]\n",
"family_examples"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build a robust class-comparison profile\n",
"\n",
"Use the median and interquartile range (IQR) because security features are often sparse and highly skewed. The standardized median gap below is an exploratory ranking only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"benign = X.loc[y == 0]\n",
"malware = X.loc[y == 1]\n",
"q75 = X.quantile(0.75)\n",
"q25 = X.quantile(0.25)\n",
"iqr = (q75 - q25).replace(0, np.nan)\n",
"\n",
"feature_profile = pd.DataFrame({\n",
" \"missing_rate\": X.isna().mean(),\n",
" \"unique_values\": X.nunique(dropna=True),\n",
" \"benign_median\": benign.median(),\n",
" \"malware_median\": malware.median(),\n",
"})\n",
"feature_profile[\"standardized_median_gap\"] = (\n",
" (feature_profile[\"malware_median\"] - feature_profile[\"benign_median\"]).abs() / iqr\n",
")\n",
"feature_profile = feature_profile.replace([np.inf, -np.inf], np.nan)\n",
"top_features = feature_profile[\"standardized_median_gap\"].nlargest(8).index.tolist()\n",
"display(feature_profile.loc[top_features])"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Visualize class medians for the strongest exploratory differences"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"median_plot = feature_profile.loc[top_features, [\"benign_median\", \"malware_median\"]]\n",
"ax = median_plot.plot(kind=\"barh\", figsize=(10, 6))\n",
"ax.set_title(\"Class medians for exploratory high-gap features\")\n",
"ax.set_xlabel(\"Median feature value\")\n",
"ax.set_ylabel(\"Feature\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A large class difference does not prove that a feature causes malicious behavior. It may reflect packers, compiler versions, collection procedures, or historical artifacts. Later models use a fixed train-test split and pipelines; this full-dataset EDA is not used to select a final test-optimized feature subset."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert set(X.columns) == set(feature_cols)\n",
"assert set(y.unique()) == {0, 1}\n",
"assert len(feature_profile) == len(feature_cols)\n",
"assert len(top_features) == 8\n",
"assert set(top_features).issubset(feature_cols)\n",
"assert {\"benign_median\", \"malware_median\", \"standardized_median_gap\"}.issubset(feature_profile.columns)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why use medians and IQR?** They are less sensitive to extreme values and skew than means and standard deviations.\n",
"2. **Can the ranking prove causation?** No. It identifies associations in this particular collection.\n",
"3. **Why not choose final features from the full table?** Doing so lets test-set patterns influence model design and makes the final score optimistic."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/03_Decision_Tree_Baseline_Solution.ipynb
+213
View File
@@ -0,0 +1,213 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 3: Building a Decision Tree Baseline\n",
"\n",
" \n",
"**Topics:** stratified split, dummy baseline, bounded tree, leakage-safe pipeline\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- create a reproducible stratified train-test split;\n",
"- fit missing-value imputation on training data only;\n",
"- compare a bounded Decision Tree with a dummy baseline;\n",
"- evaluate malware precision, recall, F1, false positives, and false negatives;\n",
"- explain why trees do not require feature scaling."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"y = df[\"label\"].astype(int)\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, stratify=y, random_state=42\n",
")\n",
"print(\"Train shape:\", X_train.shape, \"Test shape:\", X_test.shape)\n",
"print(\"Malware rate — train/test:\", y_train.mean().round(4), y_test.mean().round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Establish a non-learning baseline\n",
"\n",
"Because malware is the majority class, a most-frequent classifier can achieve high accuracy while providing little operational value."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"dummy = DummyClassifier(strategy=\"most_frequent\")\n",
"dummy.fit(X_train, y_train)\n",
"dummy_predictions = dummy.predict(X_test)\n",
"dummy_metrics = security_metrics(y_test, dummy_predictions)\n",
"pd.Series(dummy_metrics, name=\"dummy\")"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Train a bounded Decision Tree\n",
"\n",
"The imputer is inside the pipeline, so its medians are learned from training rows only. A shallow tree is intentionally chosen for readability and to reduce overfitting."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"tree_pipeline = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=4,\n",
" min_samples_leaf=10,\n",
" random_state=42,\n",
" )),\n",
"])\n",
"tree_pipeline.fit(X_train, y_train)\n",
"tree_predictions = tree_pipeline.predict(X_test)\n",
"tree_metrics = security_metrics(y_test, tree_predictions)\n",
"\n",
"comparison = pd.DataFrame([dummy_metrics, tree_metrics], index=[\"Dummy\", \"Decision Tree\"])\n",
"display(comparison.round(4))\n",
"display(pd.DataFrame(\n",
" confusion_matrix(y_test, tree_predictions, labels=[0, 1]),\n",
" index=[\"Actual benign\", \"Actual malware\"],\n",
" columns=[\"Predicted benign\", \"Predicted malware\"],\n",
"))"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"assert X_train.shape[0] + X_test.shape[0] == len(df)\n",
"assert abs(y_train.mean() - y_test.mean()) < 0.01\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 4\n",
"assert set(comparison.index) == {\"Dummy\", \"Decision Tree\"}\n",
"assert comparison.loc[\"Decision Tree\", \"malware_f1\"] >= comparison.loc[\"Dummy\", \"malware_f1\"]\n",
"print(\"✓ Lab 3 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why can dummy accuracy look high?** Malware is the majority class, so always predicting malware is often correct but creates many false positives.\n",
"2. **Why no scaler?** Decision Trees compare one feature with a threshold; multiplying a feature by a constant changes the threshold but not the ordering of samples.\n",
"3. **Why bound the depth?** A shallow tree is easier to audit and is less likely to memorize small training regions."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/04_Security_Metrics_Error_Costs_Solution.ipynb
+271
View File
@@ -0,0 +1,271 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 4: Security Metrics and Error Costs\n",
"\n",
"\n",
"**Topics:** false negatives, class weights, validation threshold, test discipline\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- quantify false-negative and false-positive costs;\n",
"- compare class-weight policies on validation data;\n",
"- select a probability threshold without inspecting the test set;\n",
"- evaluate the chosen policy once on held-out test data;\n",
"- describe the operational trade-off between missed malware and analyst workload."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"y = df[\"label\"].astype(int)\n",
"\n",
"X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
" X, y, test_size=0.20, stratify=y, random_state=42\n",
")\n",
"X_train, X_valid, y_train, y_valid = train_test_split(\n",
" X_train_full, y_train_full, test_size=0.25, stratify=y_train_full, random_state=42\n",
")\n",
"print(\"Train/validation/test:\", len(X_train), len(X_valid), len(X_test))"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare class-weight policies on validation data\n",
"\n",
"Weighting malware errors more heavily may reduce false negatives, but it can increase false positives. The correct choice depends on the cost of missed malware and available analyst capacity."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"weight_policies = {\n",
" \"unweighted\": None,\n",
" \"malware_x2\": {0: 1, 1: 2},\n",
" \"malware_x4\": {0: 1, 1: 4},\n",
"}\n",
"validation_rows = []\n",
"fitted_candidates = {}\n",
"\n",
"for name, weights in weight_policies.items():\n",
" model = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=4, min_samples_leaf=10, class_weight=weights, random_state=42\n",
" )),\n",
" ])\n",
" model.fit(X_train, y_train)\n",
" predictions = model.predict(X_valid)\n",
" validation_rows.append({\"policy\": name, **security_metrics(y_valid, predictions)})\n",
" fitted_candidates[name] = model\n",
"\n",
"validation_results = pd.DataFrame(validation_rows).set_index(\"policy\")\n",
"display(validation_results.round(4))\n",
"selected_policy = validation_results.sort_values(\n",
" [\"malware_recall\", \"false_positive_rate\", \"malware_f1\"],\n",
" ascending=[False, True, False],\n",
").index[0]\n",
"selected_model = fitted_candidates[selected_policy]\n",
"print(\"Selected class-weight policy:\", selected_policy)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a validation threshold\n",
"\n",
"Search thresholds on validation probabilities only. Prefer thresholds that achieve at least 98% validation malware recall; among them, choose the one with the lowest false-positive rate."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"validation_probabilities = selected_model.predict_proba(X_valid)[:, 1]\n",
"threshold_rows = []\n",
"for threshold in np.arange(0.20, 0.81, 0.05):\n",
" predictions = (validation_probabilities >= threshold).astype(int)\n",
" threshold_rows.append({\"threshold\": round(float(threshold), 2), **security_metrics(y_valid, predictions)})\n",
"\n",
"threshold_results = pd.DataFrame(threshold_rows).set_index(\"threshold\")\n",
"eligible = threshold_results[threshold_results[\"malware_recall\"] >= 0.98]\n",
"pool = eligible if not eligible.empty else threshold_results.loc[[threshold_results[\"malware_recall\"].idxmax()]]\n",
"selected_threshold = float(pool.sort_values(\n",
" [\"false_positive_rate\", \"malware_f1\"], ascending=[True, False]\n",
").index[0])\n",
"display(threshold_results.round(4))\n",
"print(\"Selected threshold:\", selected_threshold)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Refit on all training data and evaluate once on the test set\n",
"\n",
"The test set is used only after all policy decisions are fixed."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"final_model = Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=4,\n",
" min_samples_leaf=10,\n",
" class_weight=weight_policies[selected_policy],\n",
" random_state=42,\n",
" )),\n",
"])\n",
"final_model.fit(X_train_full, y_train_full)\n",
"test_probabilities = final_model.predict_proba(X_test)[:, 1]\n",
"default_predictions = (test_probabilities >= 0.50).astype(int)\n",
"tuned_predictions = (test_probabilities >= selected_threshold).astype(int)\n",
"\n",
"test_results = pd.DataFrame([\n",
" security_metrics(y_test, default_predictions),\n",
" security_metrics(y_test, tuned_predictions),\n",
"], index=[\"threshold_0.50\", f\"threshold_{selected_threshold:.2f}\"])\n",
"display(test_results.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(validation_results.index) == set(weight_policies)\n",
"assert 0.20 <= selected_threshold <= 0.80\n",
"assert len(test_probabilities) == len(y_test)\n",
"assert set(tuned_predictions).issubset({0, 1})\n",
"assert test_results.shape[0] == 2\n",
"assert final_model.named_steps[\"tree\"].get_depth() <= 4\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **Why tune on validation data?** Repeatedly choosing settings from test results turns the test set into training information.\n",
"2. **Why prioritize malware recall?** A false negative allows a malicious file to pass undetected. However, maximizing recall without an FPR constraint can overwhelm analysts.\n",
"3. **Is the chosen threshold universal?** No. It depends on prevalence, feature drift, collection methods, and operational costs."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/05_Tree_Interpretation_Overfitting_Solution.ipynb
+273
View File
@@ -0,0 +1,273 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 5: Interpreting Trees and Controlling Overfitting\n",
"\n",
" \n",
"**Topics:** tree rules, complexity, feature importance, train-test gap\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- compare bounded and unrestricted trees;\n",
"- quantify depth, number of leaves, and train-test performance gaps;\n",
"- visualize an explainable tree;\n",
"- export root-to-leaf logic as text rules;\n",
"- interpret feature importance cautiously."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.tree import export_text, plot_tree\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"y = df[\"label\"].astype(int)\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.20, stratify=y, random_state=42\n",
")\n",
"\n",
"def make_tree(max_depth, min_samples_leaf):\n",
" return Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(\n",
" max_depth=max_depth, min_samples_leaf=min_samples_leaf, random_state=42\n",
" )),\n",
" ])\n",
"\n",
"bounded_tree = make_tree(max_depth=4, min_samples_leaf=10)\n",
"unrestricted_tree = make_tree(max_depth=None, min_samples_leaf=1)\n",
"bounded_tree.fit(X_train, y_train)\n",
"unrestricted_tree.fit(X_train, y_train)"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Compare model complexity and generalization"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"comparison_rows = []\n",
"for name, model in {\"bounded\": bounded_tree, \"unrestricted\": unrestricted_tree}.items():\n",
" estimator = model.named_steps[\"tree\"]\n",
" train_metrics = security_metrics(y_train, model.predict(X_train))\n",
" test_metrics = security_metrics(y_test, model.predict(X_test))\n",
" comparison_rows.append({\n",
" \"model\": name,\n",
" \"depth\": estimator.get_depth(),\n",
" \"leaves\": estimator.get_n_leaves(),\n",
" \"train_f1\": train_metrics[\"malware_f1\"],\n",
" \"test_f1\": test_metrics[\"malware_f1\"],\n",
" \"f1_gap\": train_metrics[\"malware_f1\"] - test_metrics[\"malware_f1\"],\n",
" \"test_recall\": test_metrics[\"malware_recall\"],\n",
" \"test_fpr\": test_metrics[\"false_positive_rate\"],\n",
" })\n",
"complexity_comparison = pd.DataFrame(comparison_rows).set_index(\"model\")\n",
"display(complexity_comparison.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Visualize and export the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"imputer = bounded_tree.named_steps[\"imputer\"]\n",
"estimator = bounded_tree.named_steps[\"tree\"]\n",
"transformed_names = imputer.get_feature_names_out(feature_cols)\n",
"\n",
"plt.figure(figsize=(24, 12))\n",
"plot_tree(\n",
" estimator,\n",
" feature_names=transformed_names,\n",
" class_names=[\"benign\", \"malware\"],\n",
" filled=True,\n",
" rounded=True,\n",
" proportion=True,\n",
" fontsize=7,\n",
")\n",
"plt.title(\"Bounded malware Decision Tree\")\n",
"plt.tight_layout()\n",
"plt.show()\n",
"\n",
"rule_text = export_text(estimator, feature_names=list(transformed_names), max_depth=4)\n",
"print(rule_text)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Inspect impurity-based feature importance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"importance = pd.Series(\n",
" estimator.feature_importances_, index=transformed_names, name=\"importance\"\n",
").sort_values(ascending=False)\n",
"top_importance = importance.head(12)\n",
"display(top_importance.to_frame())\n",
"\n",
"ax = top_importance.sort_values().plot(kind=\"barh\", figsize=(9, 6), color=\"#4C78A8\")\n",
"ax.set_title(\"Top bounded-tree feature importances\")\n",
"ax.set_xlabel(\"Mean decrease in impurity\")\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"A rule describes this fitted tree, not a universal malware signature. Correlated features can substitute for one another, high-cardinality features may receive inflated impurity importance, and attackers may deliberately modify static properties. Treat rules as analyst evidence that requires validation."
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-013",
"metadata": {},
"outputs": [],
"source": [
"assert bounded_tree.named_steps[\"tree\"].get_depth() <= 4\n",
"assert unrestricted_tree.named_steps[\"tree\"].get_depth() >= bounded_tree.named_steps[\"tree\"].get_depth()\n",
"assert complexity_comparison.loc[\"unrestricted\", \"leaves\"] >= complexity_comparison.loc[\"bounded\", \"leaves\"]\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-014",
"metadata": {},
"source": [
"## Suggested answers\n",
"\n",
"1. **What indicates overfitting?** The unrestricted tree has more depth/leaves and typically a larger train-test performance gap.\n",
"2. **How should a rule be written?** Combine each split along one root-to-leaf path with `AND`, then attach the leaf prediction and sample support.\n",
"3. **Why is importance not causality?** It measures impurity reduction inside one fitted model and can reflect correlation or collection artifacts."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/06_Capstone_Malware_Detection_Solution.ipynb
+373
View File
@@ -0,0 +1,373 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 5 — Lab 6: Capstone — An Explainable Malware Detection Workflow\n",
"\n",
" \n",
"**Topics:** end-to-end workflow, model comparison, error evidence, analyst recommendation\n",
"\n",
"> **Solution version:** Run the cells in order, compare the outputs with your work, and read the suggested answers."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build and report a malware detector using the prepared UCI feature table. Compare an explainable bounded tree with a deeper tree and a Random Forest, then recommend a model based on detection quality, false positives, interpretability, and dataset limitations.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/capstone_metrics.csv`\n",
"- `output/capstone_errors.csv`\n",
"- `output/capstone_tree_rules.txt`\n",
"- `output/capstone_confusion_matrix.png`\n",
"- `output/capstone_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/malware_features.csv\")\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" \"Missing data/malware_features.csv. Follow data/README.md, place the three \"\n",
" \"official UCI CSV files under data/raw/, and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.tree import export_text\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"malware_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"malware_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"malware_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(DATA_PATH, low_memory=False)\n",
"feature_cols = [column for column in df.columns if column not in {\"source\", \"label\"}]\n",
"X = df[feature_cols].apply(pd.to_numeric, errors=\"coerce\")\n",
"y = df[\"label\"].astype(int)\n",
"X_train, X_test, y_train, y_test, source_train, source_test = train_test_split(\n",
" X, y, df[\"source\"], test_size=0.20, stratify=y, random_state=42\n",
")\n",
"print(\"Train/test rows:\", len(X_train), len(X_test))"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Build and compare candidate models"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"models = {\n",
" \"Dummy\": DummyClassifier(strategy=\"most_frequent\"),\n",
" \"ExplainableTree\": Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(max_depth=4, min_samples_leaf=10, random_state=42)),\n",
" ]),\n",
" \"DeepTree\": Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"tree\", DecisionTreeClassifier(random_state=42)),\n",
" ]),\n",
" \"RandomForest\": Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"median\", add_indicator=True)),\n",
" (\"forest\", RandomForestClassifier(\n",
" n_estimators=200,\n",
" min_samples_leaf=2,\n",
" random_state=42,\n",
" n_jobs=-1,\n",
" )),\n",
" ]),\n",
"}\n",
"\n",
"metric_rows = []\n",
"predictions = {}\n",
"probabilities = {}\n",
"for name, model in models.items():\n",
" model.fit(X_train, y_train)\n",
" pred = model.predict(X_test)\n",
" predictions[name] = pred\n",
" if hasattr(model, \"predict_proba\"):\n",
" probabilities[name] = model.predict_proba(X_test)[:, 1]\n",
" metric_rows.append({\"model\": name, **security_metrics(y_test, pred)})\n",
"\n",
"metrics_table = pd.DataFrame(metric_rows).set_index(\"model\").sort_values(\n",
" [\"malware_recall\", \"malware_f1\"], ascending=False\n",
")\n",
"display(metrics_table.round(4))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Select a model with an explicit decision rule\n",
"\n",
"Prefer the explainable tree when its malware recall is within 0.03 of the best non-dummy recall and its false-positive rate is no more than 0.05 above the best non-dummy FPR. Otherwise choose the non-dummy model with the highest malware F1."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"eligible_metrics = metrics_table.drop(index=\"Dummy\")\n",
"best_recall = eligible_metrics[\"malware_recall\"].max()\n",
"best_fpr = eligible_metrics[\"false_positive_rate\"].min()\n",
"explainable_ok = (\n",
" eligible_metrics.loc[\"ExplainableTree\", \"malware_recall\"] >= best_recall - 0.03\n",
" and eligible_metrics.loc[\"ExplainableTree\", \"false_positive_rate\"] <= best_fpr + 0.05\n",
")\n",
"selected_name = (\n",
" \"ExplainableTree\"\n",
" if explainable_ok\n",
" else eligible_metrics[\"malware_f1\"].idxmax()\n",
")\n",
"selected_model = models[selected_name]\n",
"selected_predictions = predictions[selected_name]\n",
"print(\"Selected model:\", selected_name)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Create error evidence and a confusion matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"selected_probability = probabilities.get(selected_name, selected_predictions.astype(float))\n",
"error_table = pd.DataFrame({\n",
" \"source\": source_test.to_numpy(),\n",
" \"true_label\": y_test.to_numpy(),\n",
" \"predicted_label\": selected_predictions,\n",
" \"malware_probability\": selected_probability,\n",
"}, index=X_test.index)\n",
"error_table[\"error_type\"] = np.select(\n",
" [\n",
" (error_table[\"true_label\"] == 1) & (error_table[\"predicted_label\"] == 0),\n",
" (error_table[\"true_label\"] == 0) & (error_table[\"predicted_label\"] == 1),\n",
" ],\n",
" [\"false_negative\", \"false_positive\"],\n",
" default=\"correct\",\n",
")\n",
"errors_only = error_table[error_table[\"error_type\"] != \"correct\"].copy()\n",
"display(errors_only[\"error_type\"].value_counts())\n",
"\n",
"fig, ax = plt.subplots(figsize=(6, 5))\n",
"ConfusionMatrixDisplay.from_predictions(\n",
" y_test,\n",
" selected_predictions,\n",
" labels=[0, 1],\n",
" display_labels=[\"benign\", \"malware\"],\n",
" cmap=\"Blues\",\n",
" ax=ax,\n",
" colorbar=False,\n",
")\n",
"ax.set_title(f\"{selected_name} — held-out test set\")\n",
"fig.tight_layout()\n",
"fig.savefig(OUTPUT_DIR / \"capstone_confusion_matrix.png\", dpi=160)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export rules and write the analyst recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"explainable_model = models[\"ExplainableTree\"]\n",
"explainable_imputer = explainable_model.named_steps[\"imputer\"]\n",
"explainable_estimator = explainable_model.named_steps[\"tree\"]\n",
"explainable_names = explainable_imputer.get_feature_names_out(feature_cols)\n",
"tree_rules = export_text(\n",
" explainable_estimator,\n",
" feature_names=list(explainable_names),\n",
" max_depth=4,\n",
")\n",
"\n",
"selected_metrics = metrics_table.loc[selected_name]\n",
"analyst_summary = \"\\n\".join([\n",
" \"# Malware Detection Capstone — Analyst Summary\",\n",
" \"\",\n",
" \"## Recommendation\",\n",
" \"\",\n",
" f\"Selected model: **{selected_name}**.\",\n",
" \"\",\n",
" f\"- Malware recall: {selected_metrics['malware_recall']:.3f}\",\n",
" f\"- Malware precision: {selected_metrics['malware_precision']:.3f}\",\n",
" f\"- Malware F1: {selected_metrics['malware_f1']:.3f}\",\n",
" f\"- False positives: {int(selected_metrics['false_positives'])}\",\n",
" f\"- False negatives: {int(selected_metrics['false_negatives'])}\",\n",
" \"\",\n",
" \"The selection followed the documented recall/FPR/interpretability rule. \"\n",
" \"The bounded tree is preferred when its detection quality is close to the \"\n",
" \"strongest candidate; otherwise the best non-dummy F1 model is selected.\",\n",
" \"\",\n",
" \"## Operational interpretation\",\n",
" \"\",\n",
" \"False negatives represent malware that would pass the detector. False positives \"\n",
" \"represent benign files that would create review workload. A deployment threshold \"\n",
" \"must therefore be matched to analyst capacity and the cost of a missed infection.\",\n",
" \"\",\n",
" \"## Limitations\",\n",
" \"\",\n",
" \"This historical dataset is imbalanced, and benign/malware records come from \"\n",
" \"different source collections. Random-split performance may include collection \"\n",
" \"artifacts and does not demonstrate robustness to future malware, concept drift, \"\n",
" \"packing, or adversarial evasion. The model should be used as one triage signal \"\n",
" \"alongside signatures, sandboxing, and analyst review.\",\n",
" \"\",\n",
"])\n",
"\n",
"metrics_table.to_csv(OUTPUT_DIR / \"capstone_metrics.csv\")\n",
"errors_only.to_csv(OUTPUT_DIR / \"capstone_errors.csv\", index=True)\n",
"(OUTPUT_DIR / \"capstone_tree_rules.txt\").write_text(tree_rules, encoding=\"utf-8\")\n",
"(OUTPUT_DIR / \"capstone_analyst_summary.md\").write_text(analyst_summary, encoding=\"utf-8\")\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"capstone_metrics.csv\",\n",
" \"capstone_errors.csv\",\n",
" \"capstone_tree_rules.txt\",\n",
" \"capstone_confusion_matrix.png\",\n",
" \"capstone_analyst_summary.md\",\n",
"}\n",
"assert set(models) == {\"Dummy\", \"ExplainableTree\", \"DeepTree\", \"RandomForest\"}\n",
"assert selected_name in set(models) - {\"Dummy\"}\n",
"assert len(error_table) == len(y_test)\n",
"assert set(errors_only[\"error_type\"].unique()).issubset({\"false_negative\", \"false_positive\"})\n",
"assert isinstance(tree_rules, str) and \"class:\" in tree_rules\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Suggested review checklist\n",
"\n",
"- The recommendation follows a rule stated before inspecting individual errors.\n",
"- Accuracy is not used as the sole decision metric.\n",
"- False negatives and false positives are translated into operational consequences.\n",
"- `source` is used only for auditing, never as a model feature.\n",
"- Tree rules are treated as model explanations, not universal malware signatures.\n",
"- Dataset age, imbalance, source confounding, drift, and evasion are acknowledged.\n",
"- The detector is positioned as one layer in a defense workflow, not an autonomous verdict."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-5/Lab-5-Solution/data/README.md
+79
View File
@@ -0,0 +1,79 @@
# Dataset setup — UCI Malware Static and Dynamic Features
All six notebooks use pre-extracted tabular features from the official UCI dataset **Malware Static and Dynamic Features VxHeaven and Virus Total**.
## Download
1. Open the [UCI dataset page](https://archive.ics.uci.edu/dataset/541/malware%2Bstatic%2Band%2Bdynamic%2Bfeatures%2Bvxheaven%2Band%2Bvirus%2Btotal).
2. Select **Download** and extract the archive.
3. Create `data/raw/` if it does not already exist.
4. Copy these three files into `data/raw/`:
```text
staDynBenignLab.csv
staDynVxHeaven2698Lab.csv
staDynVt2955Lab.csv
```
The expected layout is:
```text
Practical-5/
└── data/
├── prepare_dataset.py
└── raw/
├── staDynBenignLab.csv
├── staDynVxHeaven2698Lab.csv
└── staDynVt2955Lab.csv
```
## Prepare the classroom table
From the `Practical-5` directory, run:
```bash
python data/prepare_dataset.py
```
The utility:
- finds the three required files under `data/raw/`;
- normalizes column names and adds deterministic occurrence suffixes when
case- or punctuation-distinct UCI fields collapse to the same snake-case name;
- keeps columns shared by all three sources;
- removes obvious identifiers and pre-existing target columns;
- converts usable columns to numeric values;
- adds `source` and `label` (`0 = benign`, `1 = malware`);
- retains missing values so imputation can be fitted on training data inside the notebooks;
- writes `data/malware_features.csv` and `data/feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
path = Path("data/malware_features.csv")
df = pd.read_csv(path)
print(df.shape)
print(df["label"].value_counts().sort_index())
print(df["source"].value_counts())
```
The original release contains approximately 6,248 rows: 595 benign records and 5,653 malware records. The exact number of retained feature columns can vary if the source archive changes column formatting.
## Provenance and license
- Dataset: Malware Static and Dynamic Features VxHeaven and Virus Total
- Source: UCI Machine Learning Repository
- DOI: [10.24432/C58K6H](https://doi.org/10.24432/C58K6H)
- License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
Retain the citation and license when redistributing derived classroom tables.
## Important limitations
- The class distribution is strongly imbalanced.
- Benign and malware samples originate from different collections, so source-specific artifacts may inflate random-split performance.
- The features represent a historical dataset and should not be treated as a current production benchmark.
- No raw malware is required or permitted for this practical.
Lab-5/Lab-5-Solution/data/feature_manifest.csv
+245
View File
@@ -0,0 +1,245 @@
feature,non_missing_ratio,unique_values,feature_family
add,1.0,2809,unclassified
addressofentrypoint,1.0,3156,unclassified
align,1.0,11,unclassified
arg,1.0,5,unclassified
asc,1.0,3,unclassified
baseofcode,1.0,145,unclassified
bool,1.0,2,unclassified
bt,1.0,114,unclassified
byte,1.0,17,unclassified
call,1.0,2131,unclassified
cexit,1.0,2,unclassified
char,1.0,4,unclassified
characteristics,1.0,52,unclassified
checksum,1.0,2100,unclassified
cld,1.0,685,unclassified
cli,1.0,625,unclassified
closehandle,1.0,2,unclassified
cmc,1.0,635,unclassified
cmp,1.0,1954,unclassified
compile_date,1.0,3912,unclassified
cookie,1.0,2,unclassified
copyfilea,1.0,2,unclassified
count_dll_loaded,1.0,60,unclassified
count_file_copied,1.0,47,unclassified
count_file_deleted,1.0,19,unclassified
count_file_exists,1.0,75,unclassified
count_file_opened,1.0,76,unclassified
count_file_read,1.0,29,unclassified
count_file_written,1.0,24,unclassified
count_mutex,1.0,18,unclassified
count_regkey_deleted,1.0,12,unclassified
count_regkey_written,1.0,45,unclassified
createdirectorya,1.0,2,unclassified
createfilea,1.0,2,unclassified
createmutexa,1.0,2,unclassified
createprocessa,1.0,2,unclassified
createthread,1.0,2,unclassified
createtoolhelp32snapshot,1.0,2,unclassified
cs,1.0,529,unclassified
daa,1.0,618,unclassified
data,1.0,4,unclassified
data__2,1.0,2,unclassified
db,1.0,1827,unclassified
db_1,1.0,1147,unclassified
dd,1.0,3111,unclassified
dd_1,1.0,946,unclassified
dec,1.0,1627,unclassified
deletefilea,1.0,2,unclassified
dll,1.0,225,unclassified
dll_characteristics,1.0,27,unclassified
ds,1.0,526,unclassified
dw,1.0,63,unclassified
enablewindow,1.0,2,unclassified
ent_max,1.0,5674,unclassified
ent_mean,1.0,6243,unclassified
ent_median,1.0,6087,unclassified
ent_min,1.0,5699,unclassified
ent_var,1.0,6114,unclassified
ent_whole_file,1.0,6248,unclassified
entry,1.0,5,unclassified
es,1.0,500,unclassified
exit,1.0,33,unclassified
exitprocess,1.0,3,unclassified
exitwindowsex,1.0,2,unclassified
faddp,1.0,131,unclassified
fchs,1.0,55,unclassified
fclose,1.0,2,unclassified
fdiv,1.0,175,unclassified
fdivp,1.0,130,unclassified
fdivr,1.0,192,unclassified
fild,1.0,276,unclassified
file_alignment,1.0,5,unclassified
files_operations,1.0,159,unclassified
filesize,1.0,5252,unclassified
findclose,1.0,2,unclassified
findfirstfilea,1.0,2,unclassified
findnextfilea,1.0,2,unclassified
findwindowa,1.0,2,unclassified
fistp,1.0,202,unclassified
fld,1.0,144,unclassified
fopen,1.0,3,unclassified
free,1.0,169,unclassified
fs,1.0,683,unclassified
fstcw,1.0,19,unclassified
fwrite,1.0,3,unclassified
fxch,1.0,148,unclassified
getcommandlinea,1.0,2,unclassified
getfileattributesa,1.0,2,unclassified
getforegroundwindow,1.0,2,unclassified
getlasterror,1.0,4,unclassified
getmainargs,1.0,2,unclassified
getmodulefilenamea,1.0,3,unclassified
getmodulehandlea,1.0,3,unclassified
getstartupinfoa,1.0,2,unclassified
getsystemdirectorya,1.0,2,unclassified
getsystemmetrics,1.0,2,unclassified
gettickcount,1.0,2,unclassified
gs,1.0,796,unclassified
handle,1.0,10,unclassified
image_base,1.0,44,unclassified
imul,1.0,1113,unclassified
in,1.0,1016,unclassified
inc,1.0,1941,unclassified
initterm,1.0,2,unclassified
int,1.0,1500,unclassified
int_1,1.0,575,unclassified
interlockedcompareexchange,1.0,2,unclassified
interlockeddecrement,1.0,2,unclassified
interlockedincrement,1.0,2,unclassified
jb,1.0,1089,unclassified
je,1.0,1732,unclassified
jg,1.0,765,unclassified
jge,1.0,748,unclassified
jl,1.0,825,unclassified
jmp,1.0,1707,unclassified
jno,1.0,650,unclassified
jo,1.0,819,unclassified
lea,1.0,1673,unclassified
loader_flags,1.0,4,unclassified
loc,1.0,688,unclassified
long,1.0,10,unclassified
loope,1.0,580,unclassified
lstrcata,1.0,2,unclassified
lstrcpya,1.0,2,unclassified
lstrlena,1.0,2,unclassified
machine,1.0,2,unclassified
magic,1.0,2,unclassified
major_image_version,1.0,39,unclassified
major_linker_version,1.0,30,unclassified
major_operating_system_version,1.0,7,unclassified
major_subsystem_version,1.0,5,unclassified
malloc,1.0,5,unclassified
max_min,1.0,5517,unclassified
memcpy,1.0,5,unclassified
memmove,1.0,3,unclassified
memset,1.0,5,unclassified
messageboxa,1.0,2,unclassified
minor_image_version,1.0,44,unclassified
minor_linker_version,1.0,47,unclassified
minor_operating_system_version,1.0,10,unclassified
minor_subsystem_version,1.0,6,unclassified
mov,1.0,3156,unclassified
mul,1.0,207,unclassified
neg,1.0,301,unclassified
nop,1.0,1037,unclassified
not,1.0,265,unclassified
number_of_iat_entires,1.0,467,unclassified
number_of_iat_entires_1,1.0,467,unclassified
number_of_imports_1,1.0,40,unclassified
number_of_rva_and_sizes,1.0,8,unclassified
number_of_rva_and_sizes_1,1.0,8,unclassified
number_of_sections_1,1.0,29,unclassified
number_of_symbols,1.0,50,unclassified
off,1.0,4,unclassified
or,1.0,1497,unclassified
out,1.0,1013,unclassified
p_fmode,1.0,2,unclassified
pointer_to_symbol_table,1.0,42,unclassified
pop,1.0,2306,unclassified
popf,1.0,585,unclassified
proc,1.0,2,unclassified
process32first,1.0,2,unclassified
process32next,1.0,2,unclassified
ptr,1.0,16,unclassified
push,1.0,3002,unclassified
pushf,1.0,642,unclassified
raiseexception,1.0,3,unclassified
rand,1.0,6,unclassified
rcl,1.0,314,unclassified
rcr,1.0,261,unclassified
rdata,1.0,2,unclassified
rdtsc,1.0,50,unclassified
regclosekey,1.0,3,unclassified
regopenkeya,1.0,2,unclassified
regopenkeyexa,1.0,2,unclassified
regsetvalueexa,1.0,3,unclassified
rep,1.0,263,unclassified
ret,1.0,1458,unclassified
rol,1.0,348,unclassified
ror,1.0,332,unclassified
sar,1.0,466,unclassified
sbb,1.0,1230,unclassified
scas,1.0,777,unclassified
section_alignment,1.0,5,unclassified
sendmessagea,1.0,2,unclassified
set_app_type,1.0,3,unclassified
setb,1.0,54,unclassified
setcursorpos,1.0,2,unclassified
setfileattributesa,1.0,3,unclassified
setlasterror,1.0,2,unclassified
setle,1.0,50,unclassified
setusermatherr,1.0,2,unclassified
setwindowtexta,1.0,2,unclassified
shellexecutea,1.0,2,unclassified
shl,1.0,493,unclassified
shld,1.0,79,unclassified
showwindow,1.0,2,unclassified
shr,1.0,499,unclassified
sidt,1.0,2,unclassified
size_code,1.0,930,unclassified
size_image,1.0,829,unclassified
size_init_data,1.0,948,unclassified
size_of_headers,1.0,30,unclassified
size_of_heap_commit,1.0,10,unclassified
size_of_heap_reserve,1.0,26,unclassified
size_of_optional_header,1.0,2,unclassified
size_of_stack_commit,1.0,35,unclassified
size_of_stack_reserve,1.0,34,unclassified
size_uninit_data,1.0,164,unclassified
sizeofheaders,1.0,30,unclassified
sizeofheaders_1,1.0,30,unclassified
sleep,1.0,4,unclassified
sprintf,1.0,2,unclassified
ss,1.0,577,unclassified
stc,1.0,634,unclassified
std,1.0,658,unclassified
sti,1.0,669,unclassified
stos,1.0,790,unclassified
strcat,1.0,4,unclassified
strchr,1.0,2,unclassified
strcmp,1.0,4,unclassified
strcpy,1.0,4,unclassified
strlen,1.0,6,unclassified
strncmp,1.0,2,unclassified
strstr,1.0,3,unclassified
sub,1.0,1631,unclassified
subsystem,1.0,3,unclassified
test,1.0,1685,unclassified
text,1.0,5,unclassified
time,1.0,5,unclassified
tlsalloc,1.0,2,unclassified
tlsfree,1.0,2,unclassified
tlsgetvalue,1.0,2,unclassified
tlssetvalue,1.0,2,unclassified
uint,1.0,3,unclassified
unhandledexceptionfilter,1.0,2,unclassified
waitforsingleobject,1.0,2,unclassified
winexec,1.0,2,unclassified
winmain,1.0,6,unclassified
word,1.0,9,unclassified
word__2,1.0,3,unclassified
writefile,1.0,2,unclassified
xchg,1.0,1217,unclassified
xor,1.0,1887,unclassified
1 feature non_missing_ratio unique_values feature_family
2 add 1.0 2809 unclassified
3 addressofentrypoint 1.0 3156 unclassified
4 align 1.0 11 unclassified
5 arg 1.0 5 unclassified
6 asc 1.0 3 unclassified
7 baseofcode 1.0 145 unclassified
8 bool 1.0 2 unclassified
9 bt 1.0 114 unclassified
10 byte 1.0 17 unclassified
11 call 1.0 2131 unclassified
12 cexit 1.0 2 unclassified
13 char 1.0 4 unclassified
14 characteristics 1.0 52 unclassified
15 checksum 1.0 2100 unclassified
16 cld 1.0 685 unclassified
17 cli 1.0 625 unclassified
18 closehandle 1.0 2 unclassified
19 cmc 1.0 635 unclassified
20 cmp 1.0 1954 unclassified
21 compile_date 1.0 3912 unclassified
22 cookie 1.0 2 unclassified
23 copyfilea 1.0 2 unclassified
24 count_dll_loaded 1.0 60 unclassified
25 count_file_copied 1.0 47 unclassified
26 count_file_deleted 1.0 19 unclassified
27 count_file_exists 1.0 75 unclassified
28 count_file_opened 1.0 76 unclassified
29 count_file_read 1.0 29 unclassified
30 count_file_written 1.0 24 unclassified
31 count_mutex 1.0 18 unclassified
32 count_regkey_deleted 1.0 12 unclassified
33 count_regkey_written 1.0 45 unclassified
34 createdirectorya 1.0 2 unclassified
35 createfilea 1.0 2 unclassified
36 createmutexa 1.0 2 unclassified
37 createprocessa 1.0 2 unclassified
38 createthread 1.0 2 unclassified
39 createtoolhelp32snapshot 1.0 2 unclassified
40 cs 1.0 529 unclassified
41 daa 1.0 618 unclassified
42 data 1.0 4 unclassified
43 data__2 1.0 2 unclassified
44 db 1.0 1827 unclassified
45 db_1 1.0 1147 unclassified
46 dd 1.0 3111 unclassified
47 dd_1 1.0 946 unclassified
48 dec 1.0 1627 unclassified
49 deletefilea 1.0 2 unclassified
50 dll 1.0 225 unclassified
51 dll_characteristics 1.0 27 unclassified
52 ds 1.0 526 unclassified
53 dw 1.0 63 unclassified
54 enablewindow 1.0 2 unclassified
55 ent_max 1.0 5674 unclassified
56 ent_mean 1.0 6243 unclassified
57 ent_median 1.0 6087 unclassified
58 ent_min 1.0 5699 unclassified
59 ent_var 1.0 6114 unclassified
60 ent_whole_file 1.0 6248 unclassified
61 entry 1.0 5 unclassified
62 es 1.0 500 unclassified
63 exit 1.0 33 unclassified
64 exitprocess 1.0 3 unclassified
65 exitwindowsex 1.0 2 unclassified
66 faddp 1.0 131 unclassified
67 fchs 1.0 55 unclassified
68 fclose 1.0 2 unclassified
69 fdiv 1.0 175 unclassified
70 fdivp 1.0 130 unclassified
71 fdivr 1.0 192 unclassified
72 fild 1.0 276 unclassified
73 file_alignment 1.0 5 unclassified
74 files_operations 1.0 159 unclassified
75 filesize 1.0 5252 unclassified
76 findclose 1.0 2 unclassified
77 findfirstfilea 1.0 2 unclassified
78 findnextfilea 1.0 2 unclassified
79 findwindowa 1.0 2 unclassified
80 fistp 1.0 202 unclassified
81 fld 1.0 144 unclassified
82 fopen 1.0 3 unclassified
83 free 1.0 169 unclassified
84 fs 1.0 683 unclassified
85 fstcw 1.0 19 unclassified
86 fwrite 1.0 3 unclassified
87 fxch 1.0 148 unclassified
88 getcommandlinea 1.0 2 unclassified
89 getfileattributesa 1.0 2 unclassified
90 getforegroundwindow 1.0 2 unclassified
91 getlasterror 1.0 4 unclassified
92 getmainargs 1.0 2 unclassified
93 getmodulefilenamea 1.0 3 unclassified
94 getmodulehandlea 1.0 3 unclassified
95 getstartupinfoa 1.0 2 unclassified
96 getsystemdirectorya 1.0 2 unclassified
97 getsystemmetrics 1.0 2 unclassified
98 gettickcount 1.0 2 unclassified
99 gs 1.0 796 unclassified
100 handle 1.0 10 unclassified
101 image_base 1.0 44 unclassified
102 imul 1.0 1113 unclassified
103 in 1.0 1016 unclassified
104 inc 1.0 1941 unclassified
105 initterm 1.0 2 unclassified
106 int 1.0 1500 unclassified
107 int_1 1.0 575 unclassified
108 interlockedcompareexchange 1.0 2 unclassified
109 interlockeddecrement 1.0 2 unclassified
110 interlockedincrement 1.0 2 unclassified
111 jb 1.0 1089 unclassified
112 je 1.0 1732 unclassified
113 jg 1.0 765 unclassified
114 jge 1.0 748 unclassified
115 jl 1.0 825 unclassified
116 jmp 1.0 1707 unclassified
117 jno 1.0 650 unclassified
118 jo 1.0 819 unclassified
119 lea 1.0 1673 unclassified
120 loader_flags 1.0 4 unclassified
121 loc 1.0 688 unclassified
122 long 1.0 10 unclassified
123 loope 1.0 580 unclassified
124 lstrcata 1.0 2 unclassified
125 lstrcpya 1.0 2 unclassified
126 lstrlena 1.0 2 unclassified
127 machine 1.0 2 unclassified
128 magic 1.0 2 unclassified
129 major_image_version 1.0 39 unclassified
130 major_linker_version 1.0 30 unclassified
131 major_operating_system_version 1.0 7 unclassified
132 major_subsystem_version 1.0 5 unclassified
133 malloc 1.0 5 unclassified
134 max_min 1.0 5517 unclassified
135 memcpy 1.0 5 unclassified
136 memmove 1.0 3 unclassified
137 memset 1.0 5 unclassified
138 messageboxa 1.0 2 unclassified
139 minor_image_version 1.0 44 unclassified
140 minor_linker_version 1.0 47 unclassified
141 minor_operating_system_version 1.0 10 unclassified
142 minor_subsystem_version 1.0 6 unclassified
143 mov 1.0 3156 unclassified
144 mul 1.0 207 unclassified
145 neg 1.0 301 unclassified
146 nop 1.0 1037 unclassified
147 not 1.0 265 unclassified
148 number_of_iat_entires 1.0 467 unclassified
149 number_of_iat_entires_1 1.0 467 unclassified
150 number_of_imports_1 1.0 40 unclassified
151 number_of_rva_and_sizes 1.0 8 unclassified
152 number_of_rva_and_sizes_1 1.0 8 unclassified
153 number_of_sections_1 1.0 29 unclassified
154 number_of_symbols 1.0 50 unclassified
155 off 1.0 4 unclassified
156 or 1.0 1497 unclassified
157 out 1.0 1013 unclassified
158 p_fmode 1.0 2 unclassified
159 pointer_to_symbol_table 1.0 42 unclassified
160 pop 1.0 2306 unclassified
161 popf 1.0 585 unclassified
162 proc 1.0 2 unclassified
163 process32first 1.0 2 unclassified
164 process32next 1.0 2 unclassified
165 ptr 1.0 16 unclassified
166 push 1.0 3002 unclassified
167 pushf 1.0 642 unclassified
168 raiseexception 1.0 3 unclassified
169 rand 1.0 6 unclassified
170 rcl 1.0 314 unclassified
171 rcr 1.0 261 unclassified
172 rdata 1.0 2 unclassified
173 rdtsc 1.0 50 unclassified
174 regclosekey 1.0 3 unclassified
175 regopenkeya 1.0 2 unclassified
176 regopenkeyexa 1.0 2 unclassified
177 regsetvalueexa 1.0 3 unclassified
178 rep 1.0 263 unclassified
179 ret 1.0 1458 unclassified
180 rol 1.0 348 unclassified
181 ror 1.0 332 unclassified
182 sar 1.0 466 unclassified
183 sbb 1.0 1230 unclassified
184 scas 1.0 777 unclassified
185 section_alignment 1.0 5 unclassified
186 sendmessagea 1.0 2 unclassified
187 set_app_type 1.0 3 unclassified
188 setb 1.0 54 unclassified
189 setcursorpos 1.0 2 unclassified
190 setfileattributesa 1.0 3 unclassified
191 setlasterror 1.0 2 unclassified
192 setle 1.0 50 unclassified
193 setusermatherr 1.0 2 unclassified
194 setwindowtexta 1.0 2 unclassified
195 shellexecutea 1.0 2 unclassified
196 shl 1.0 493 unclassified
197 shld 1.0 79 unclassified
198 showwindow 1.0 2 unclassified
199 shr 1.0 499 unclassified
200 sidt 1.0 2 unclassified
201 size_code 1.0 930 unclassified
202 size_image 1.0 829 unclassified
203 size_init_data 1.0 948 unclassified
204 size_of_headers 1.0 30 unclassified
205 size_of_heap_commit 1.0 10 unclassified
206 size_of_heap_reserve 1.0 26 unclassified
207 size_of_optional_header 1.0 2 unclassified
208 size_of_stack_commit 1.0 35 unclassified
209 size_of_stack_reserve 1.0 34 unclassified
210 size_uninit_data 1.0 164 unclassified
211 sizeofheaders 1.0 30 unclassified
212 sizeofheaders_1 1.0 30 unclassified
213 sleep 1.0 4 unclassified
214 sprintf 1.0 2 unclassified
215 ss 1.0 577 unclassified
216 stc 1.0 634 unclassified
217 std 1.0 658 unclassified
218 sti 1.0 669 unclassified
219 stos 1.0 790 unclassified
220 strcat 1.0 4 unclassified
221 strchr 1.0 2 unclassified
222 strcmp 1.0 4 unclassified
223 strcpy 1.0 4 unclassified
224 strlen 1.0 6 unclassified
225 strncmp 1.0 2 unclassified
226 strstr 1.0 3 unclassified
227 sub 1.0 1631 unclassified
228 subsystem 1.0 3 unclassified
229 test 1.0 1685 unclassified
230 text 1.0 5 unclassified
231 time 1.0 5 unclassified
232 tlsalloc 1.0 2 unclassified
233 tlsfree 1.0 2 unclassified
234 tlsgetvalue 1.0 2 unclassified
235 tlssetvalue 1.0 2 unclassified
236 uint 1.0 3 unclassified
237 unhandledexceptionfilter 1.0 2 unclassified
238 waitforsingleobject 1.0 2 unclassified
239 winexec 1.0 2 unclassified
240 winmain 1.0 6 unclassified
241 word 1.0 9 unclassified
242 word__2 1.0 3 unclassified
243 writefile 1.0 2 unclassified
244 xchg 1.0 1217 unclassified
245 xor 1.0 1887 unclassified
Lab-5/Lab-5-Solution/data/malware_features.csv
+6249
View File
File diff suppressed because it is too large Load Diff
Lab-5/Lab-5-Solution/data/prepare_dataset.py
+168
View File
@@ -0,0 +1,168 @@
"""Prepare a safe classroom feature table from the three UCI CSV files."""
from pathlib import Path
import re
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
OUTPUT_PATH = HERE / "malware_features.csv"
MANIFEST_PATH = HERE / "feature_manifest.csv"
FILES = {
"benign_lab": ("staDynBenignLab.csv", 0),
"vxheaven": ("staDynVxHeaven2698Lab.csv", 1),
"virustotal": ("staDynVt2955Lab.csv", 1),
}
FORBIDDEN_NAMES = {
"label", "class", "target", "malware", "is_malware", "type",
"file", "filename", "file_name", "name", "hash", "md5", "sha1",
"sha256", "source", "category",
}
def normalize_name(value: object) -> str:
"""Convert a source column name to a stable snake_case identifier."""
text = re.sub(r"[^0-9A-Za-z]+", "_", str(value).strip()).strip("_").lower()
return text or "unnamed_feature"
def normalize_columns(columns: list[object]) -> tuple[list[str], dict[str, list[str]]]:
"""Normalize names and disambiguate collisions by their source-column order.
The UCI files intentionally contain case- or punctuation-distinct fields such as
``WORD``/``word`` and ``Import``/``__IMPORT_``. A simple lowercase snake-case
conversion collapses each pair. The three source files use the same relative
order for these fields, so an occurrence suffix produces stable shared names:
``word``, ``word__2``, and so on.
"""
counts: dict[str, int] = {}
normalized: list[str] = []
originals_by_base: dict[str, list[str]] = {}
for raw_column in columns:
base = normalize_name(raw_column)
counts[base] = counts.get(base, 0) + 1
occurrence = counts[base]
unique_name = base if occurrence == 1 else f"{base}__{occurrence}"
normalized.append(unique_name)
originals_by_base.setdefault(base, []).append(str(raw_column))
collisions = {
base: originals
for base, originals in originals_by_base.items()
if len(originals) > 1
}
if len(normalized) != len(set(normalized)):
raise ValueError("Column names are still duplicated after disambiguation.")
return normalized, collisions
def find_required_file(filename: str) -> Path:
"""Find one required CSV below data/raw, allowing an extracted subfolder."""
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and check the extracted folder structure."
)
return matches[0]
def load_source(source: str, filename: str, label: int) -> pd.DataFrame:
"""Load one source file and attach provenance columns."""
frame = pd.read_csv(find_required_file(filename), low_memory=False)
normalized, collisions = normalize_columns(frame.columns.tolist())
frame.columns = normalized
# Consolidate the wide table before adding provenance columns. This avoids
# pandas fragmentation warnings on the 1,000+ column UCI source files.
frame = frame.copy()
frame.attrs["normalized_column_collisions"] = collisions
frame["source"] = source
frame["label"] = label
return frame
def validate_collision_alignment(frames: dict[str, pd.DataFrame]) -> None:
"""Ensure occurrence suffixes refer to the same raw fields in every source."""
collision_names = set().union(*(
set(frame.attrs.get("normalized_column_collisions", {}))
for frame in frames.values()
))
for base in sorted(collision_names):
variants = {
source: tuple(frame.attrs.get("normalized_column_collisions", {}).get(base, []))
for source, frame in frames.items()
}
if len(set(variants.values())) != 1:
raise ValueError(
f"Normalized collision {base!r} is not aligned across source files: {variants}"
)
if collision_names:
print("Disambiguated normalized column collisions:")
reference = next(iter(frames.values())).attrs["normalized_column_collisions"]
for base in sorted(collision_names):
generated = [base] + [f"{base}__{index}" for index in range(2, len(reference[base]) + 1)]
print(f" {reference[base]} -> {generated}")
def main() -> None:
frames = {
source: load_source(source, filename, label)
for source, (filename, label) in FILES.items()
}
validate_collision_alignment(frames)
common = set.intersection(*(set(frame.columns) for frame in frames.values()))
candidate_features = sorted(common - FORBIDDEN_NAMES - {"source", "label"})
if not candidate_features:
raise ValueError("No shared candidate feature columns were found.")
combined_parts = []
for source, frame in frames.items():
part = frame[candidate_features].copy()
part.insert(0, "source", source)
part.insert(1, "label", int(FILES[source][1]))
combined_parts.append(part)
combined = pd.concat(combined_parts, ignore_index=True)
numeric = combined[candidate_features].apply(pd.to_numeric, errors="coerce")
numeric = numeric.replace([np.inf, -np.inf], np.nan)
usable_ratio = numeric.notna().mean()
retained = usable_ratio[usable_ratio >= 0.95].index.tolist()
numeric = numeric[retained]
non_constant = numeric.nunique(dropna=True)
retained = non_constant[non_constant > 1].index.tolist()
numeric = numeric[retained]
if numeric.empty:
raise ValueError("No usable numeric features remained after validation.")
prepared = pd.concat([combined[["source", "label"]], numeric], axis=1)
prepared = prepared.sample(frac=1.0, random_state=42).reset_index(drop=True)
prepared.to_csv(OUTPUT_PATH, index=False)
manifest = pd.DataFrame({
"feature": retained,
"non_missing_ratio": numeric[retained].notna().mean().values,
"unique_values": numeric[retained].nunique(dropna=True).values,
"feature_family": "unclassified",
})
manifest.to_csv(MANIFEST_PATH, index=False)
print(f"Created {OUTPUT_PATH}")
print(f"Shape: {prepared.shape}")
print("Class counts:")
print(prepared["label"].value_counts().sort_index())
print("Source counts:")
print(prepared["source"].value_counts())
print(f"Created {MANIFEST_PATH}")
if __name__ == "__main__":
main()
Lab-5/Lab-5-Solution/data/raw/staDynBenignLab.csv
+596
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Solution/data/raw/staDynVt2955Lab.csv
+2956
View File
File diff suppressed because one or more lines are too long
Lab-5/Lab-5-Solution/data/raw/staDynVxHeaven2698Lab.csv
+2699
View File
File diff suppressed because one or more lines are too long
Lab-6/Lab-6-Practical.zip
BIN
View File
Binary file not shown.
Lab-6/Lab-6-Practical/01_NSL_KDD_Flow_Audit_Lab.ipynb
+212
View File
@@ -0,0 +1,212 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 1: Auditing NSL-KDD Flow Records\n",
"\n",
" \n",
"**Topics:** network flows, fixed split, labels, attack groups, leakage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- distinguish network-flow features from targets and audit fields;\n",
"- preserve the predefined NSL-KDD train/test split;\n",
"- describe binary labels and the DoS, Probe, R2L, and U2R groups;\n",
"- identify attack labels that appear only in the test set;\n",
"- document imbalance, duplicates, and benchmark limitations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Load and validate the fixed split"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the prepared train and test tables.\n",
"train_df = None\n",
"test_df = None\n",
"\n",
"# TODO 2: create feature_cols and numeric_features without target/audit columns.\n",
"feature_cols = None\n",
"numeric_features = None\n",
"\n",
"print(\"Train shape:\", None)\n",
"print(\"Test shape:\", None)\n",
"print(\"Model features:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"## 2. Audit classes, groups, and unseen attack labels"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-007",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: calculate binary-label and attack-group counts for both splits.\n",
"train_binary = None\n",
"test_binary = None\n",
"train_groups = None\n",
"test_groups = None\n",
"\n",
"# TODO 4: find attack labels present in test but absent from training.\n",
"train_attack_labels = None\n",
"test_attack_labels = None\n",
"unseen_test_attacks = None\n",
"\n",
"display(pd.DataFrame({\"train\": train_binary, \"test\": test_binary}))\n",
"print(\"Attack labels seen only in test:\", unseen_test_attacks)"
]
},
{
"cell_type": "markdown",
"id": "cell-008",
"metadata": {},
"source": [
"## 3. Audit quality and leakage risks"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-009",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: build a per-feature table of missing counts and unique counts.\n",
"quality_summary = None\n",
"\n",
"# TODO 6: count exact duplicate rows separately in train and test.\n",
"duplicate_counts = None\n",
"\n",
"# TODO 7: list target/audit columns that must never enter the model.\n",
"forbidden_model_columns = None\n",
"print(duplicate_counts)\n",
"print(\"Columns excluded from modeling:\", forbidden_model_columns)"
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Benchmark note\n",
"\n",
"NSL-KDD is useful for learning a reproducible IDS workflow. Its fixed test set includes attacks that are absent from training, but it remains a legacy benchmark and does not represent modern production traffic."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert len(feature_cols) == 41\n",
"assert set(CATEGORICAL_FEATURES).issubset(feature_cols)\n",
"assert set(train_df[\"binary_label\"].unique()) == {0, 1}\n",
"assert set(test_df[\"attack_group\"].unique()).issubset({\"Normal\", \"DoS\", \"Probe\", \"R2L\", \"U2R\"})\n",
"assert len(unseen_test_attacks) > 0\n",
"assert set(feature_cols).isdisjoint(TARGET_COLUMNS)\n",
"assert quality_summary.shape[0] == 41\n",
"print(\"✓ Lab 1 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why should the predefined train/test split be preserved?\n",
"2. Why must `difficulty` and derived label columns be excluded from X?\n",
"3. Why can binary accuracy hide failure on rare attack groups?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/02_Flow_Features_Preprocessing_Lab.ipynb
+205
View File
@@ -0,0 +1,205 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 2: Flow Features and Leakage-Safe Preprocessing\n",
"\n",
"\n",
"**Topics:** feature families, log transform, encoding, scaling, train-only fitting\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- interpret basic, content, traffic, and host-based flow features;\n",
"- apply `log1p` to skewed byte counts without deleting zero-byte flows;\n",
"- compare normal and attack distributions;\n",
"- build separate preprocessing plans for distance models and tree models;\n",
"- fit encoders and scalers using training data only."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load both splits and define feature_cols/numeric_features.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"\n",
"# TODO 2: create feature_families for basic, content, traffic, and host-based features.\n",
"feature_families = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Inspect byte-count skew and apply `log1p`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: create log_src_bytes and log_dst_bytes using np.log1p.\n",
"exploratory = None\n",
"\n",
"# TODO 4: calculate class medians and plot a sampled log-byte scatter chart.\n",
"byte_summary = None\n",
"sample = None\n",
"display(byte_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Build preprocessing plans for two model families"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: build supervised_preprocessor with OneHotEncoder for the three\n",
"# categorical features and median imputation for numeric features.\n",
"supervised_preprocessor = None\n",
"\n",
"distance_features = [\n",
" \"duration\", \"src_bytes\", \"dst_bytes\", \"count\", \"srv_count\",\n",
" \"serror_rate\", \"rerror_rate\", \"dst_host_count\", \"dst_host_srv_count\",\n",
"]\n",
"# TODO 6: build distance_preprocessor with median imputation and StandardScaler.\n",
"distance_preprocessor = None\n",
"\n",
"# TODO 7: fit both preprocessors on training data only and record output shapes.\n",
"supervised_shape = None\n",
"distance_shape = None\n",
"print(supervised_shape, distance_shape)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Why two plans?\n",
"\n",
"Distance and boundary methods are sensitive to feature scale, so standardization is required. A Decision Tree does not need scaling, but it does need categorical encoding and a consistent missing-value policy."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert len(feature_cols) == 41\n",
"assert exploratory[[\"log_src_bytes\", \"log_dst_bytes\"]].ge(0).all().all()\n",
"assert supervised_shape[0] == 100\n",
"assert supervised_shape[1] > len(feature_cols)\n",
"assert distance_shape == (100, len(distance_features))\n",
"assert all(feature in numeric_features for feature in distance_features)\n",
"print(\"✓ Lab 2 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is `log1p` preferable to dropping zero-byte flows?\n",
"2. Why do distance methods need scaling while Decision Trees usually do not?\n",
"3. What information leaks when a scaler or encoder is fitted on test data?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/03_kNN_Anomaly_Scoring_Lab.ipynb
+222
View File
@@ -0,0 +1,222 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 3: kNN Anomaly Scoring from Normal Traffic\n",
"\n",
"\n",
"**Topics:** normal-only baseline, nearest-neighbor distance, validation threshold, alert rate\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- train an anomaly baseline using normal traffic only;\n",
"- turn nearest-neighbor distance into an anomaly score;\n",
"- choose a threshold from held-out normal validation scores;\n",
"- evaluate attack recall and false-positive rate on the fixed test set;\n",
"- explain how threshold choice controls SOC alert volume."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.neighbors import NearestNeighbors\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, select src_bytes/dst_bytes, and split normal training\n",
"# records into 80% baseline fit and 20% validation.\n",
"train_df = None\n",
"test_df = None\n",
"byte_features = [\"src_bytes\", \"dst_bytes\"]\n",
"normal_fit, normal_valid = None, None\n",
"\n",
"# TODO 2: implement log_bytes, fit StandardScaler on normal_fit only,\n",
"# and transform fit, validation, and test records.\n",
"def log_bytes(frame):\n",
" pass\n",
"\n",
"scaler = None\n",
"X_fit = None\n",
"X_valid = None\n",
"X_test = None\n",
"y_test = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit the baseline and calculate anomaly scores"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: fit NearestNeighbors with k=5 on X_fit.\n",
"k = 5\n",
"knn = None\n",
"\n",
"# TODO 4: define the anomaly score as distance to the kth neighbor.\n",
"def knn_anomaly_score(matrix):\n",
" pass\n",
"\n",
"# TODO 5: select the 99th percentile of validation normal scores and\n",
"# evaluate thresholded predictions on the test set.\n",
"validation_scores = None\n",
"test_scores = None\n",
"threshold = None\n",
"test_predictions = None\n",
"knn_metrics = None\n",
"display(pd.Series(knn_metrics, name=\"kNN anomaly detector\"))"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Visualize scores and threshold sensitivity"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: plot normal/attack score histograms and draw the threshold.\n",
"\n",
"# TODO 7: compare 0.95, 0.975, 0.99, and 0.995 normal-score quantiles.\n",
"sensitivity = None\n",
"display(sensitivity)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"assert set(train_df.loc[normal_fit.index, \"binary_label\"].unique()) == {0}\n",
"assert len(validation_scores) == len(normal_valid)\n",
"assert len(test_scores) == len(test_df)\n",
"assert threshold == np.quantile(validation_scores, 0.99)\n",
"assert set(test_predictions).issubset({0, 1})\n",
"assert sensitivity.shape[0] == 4\n",
"print(\"✓ Lab 3 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why should attacks be excluded from the baseline fit data?\n",
"2. Why is the threshold selected from normal validation scores rather than test labels?\n",
"3. How does lowering the threshold affect missed attacks and analyst workload?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/04_OneClass_IsolationForest_Lab.ipynb
+225
View File
@@ -0,0 +1,225 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 4: One-Class SVM and Isolation Forest\n",
"\n",
" \n",
"**Topics:** normal-only learning, anomaly scores, validation thresholds, PR-AUC, runtime\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- fit two normal-only novelty detectors;\n",
"- standardize features for One-Class SVM;\n",
"- derive a common anomaly-score convention;\n",
"- compare detectors with the same normal-validation alert budget;\n",
"- explain performance and scalability trade-offs."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.svm import OneClassSVM\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load data, define anomaly_features, split normal training rows,\n",
"# and cap the normal fit sample at 5,000 records.\n",
"train_df = None\n",
"test_df = None\n",
"anomaly_features = None\n",
"normal_fit = None\n",
"normal_valid = None\n",
"X_test = None\n",
"y_test = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit the two detectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 2: build StandardScaler pipelines for OneClassSVM and IsolationForest.\n",
"detectors = None\n",
"\n",
"# TODO 3: fit both detectors on normal_fit only.\n",
"print(\"Fitted detectors:\", None)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Apply a common validation-threshold policy"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: for each detector, negate decision_function so larger means\n",
"# more anomalous, choose the 99th normal-validation percentile, and evaluate.\n",
"detector_outputs = None\n",
"detector_comparison = None\n",
"display(detector_comparison)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Compare attack-group recall"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 5: calculate recall separately for DoS, Probe, R2L, and U2R.\n",
"group_recall = None\n",
"display(group_recall)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"assert set(detectors) == {\"One-Class SVM\", \"Isolation Forest\"}\n",
"assert len(normal_fit) <= 5000\n",
"assert all(len(output[\"test_scores\"]) == len(test_df) for output in detector_outputs.values())\n",
"assert detector_comparison.shape[0] == 2\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert detector_comparison[\"false_positive_rate\"].between(0, 1).all()\n",
"print(\"✓ Lab 4 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. Why is One-Class SVM trained on a capped sample?\n",
"2. Why convert both model outputs to a common anomaly-score direction?\n",
"3. Why should per-group recall accompany aggregate attack recall?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/05_Supervised_Intrusion_Tree_Lab.ipynb
+228
View File
@@ -0,0 +1,228 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 5: A Supervised Intrusion Decision Tree\n",
"\n",
" \n",
"**Topics:** known-attack classification, fixed test set, tree rules, group recall\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Learning objectives\n",
"\n",
"After completing this lab, you will be able to:\n",
"\n",
"- train a binary classifier from labeled normal and attack records;\n",
"- apply categorical encoding inside a pipeline;\n",
"- evaluate the fixed test set without tuning on it;\n",
"- inspect Decision Tree rules and feature importance;\n",
"- compare aggregate attack recall with group-level recall."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"from sklearn.tree import DecisionTreeClassifier, export_text\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the fixed split and create X_train/y_train/X_test/y_test.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"X_train, y_train, X_test, y_test = None, None, None, None\n",
"\n",
"# TODO 2: build a ColumnTransformer and a DecisionTree pipeline using\n",
"# max_depth=8, min_samples_leaf=5, class_weight='balanced', random_state=42.\n",
"preprocessor = None\n",
"tree_pipeline = None\n",
"\n",
"# TODO 3: fit the pipeline on the fixed training split."
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Evaluate binary and attack-group performance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: predict labels/probabilities and calculate security metrics plus PR-AUC.\n",
"tree_predictions = None\n",
"tree_probabilities = None\n",
"tree_metrics = None\n",
"display(pd.Series(tree_metrics, name=\"Decision Tree\"))\n",
"\n",
"# TODO 5: calculate recall for DoS, Probe, R2L, and U2R.\n",
"group_recall = None\n",
"display(group_recall)"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Interpret the bounded tree"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: obtain transformed feature names, calculate top-12 impurity\n",
"# importances, export rules through depth 3, and plot the importance table.\n",
"fitted_preprocessor = None\n",
"fitted_tree = None\n",
"transformed_names = None\n",
"importance = None\n",
"top_importance = None\n",
"rule_text = None\n",
"print(rule_text)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## Interpretation warning\n",
"\n",
"High performance on NSL-KDD does not establish production readiness. Tree rules can reflect benchmark artifacts, and attack-group recall can expose failures hidden by the binary score."
]
},
{
"cell_type": "markdown",
"id": "cell-010",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-011",
"metadata": {},
"outputs": [],
"source": [
"assert tree_pipeline.named_steps[\"tree\"].get_depth() <= 8\n",
"assert len(tree_predictions) == len(test_df)\n",
"assert set(tree_predictions).issubset({0, 1})\n",
"assert set(group_recall.index) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert isinstance(rule_text, str) and \"class:\" in rule_text\n",
"assert np.isclose(importance.sum(), 1.0)\n",
"assert len(top_importance) <= 12\n",
"print(\"✓ Lab 5 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-012",
"metadata": {},
"source": [
"## Reflection questions\n",
"\n",
"1. How does supervised learning differ from normal-only novelty detection?\n",
"2. Which attack group is hardest for the tree, and why might binary metrics hide it?\n",
"3. Why should exported tree rules not be treated as universal IDS signatures?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/06_Capstone_Hybrid_IDS_Lab.ipynb
+288
View File
@@ -0,0 +1,288 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 6: Capstone — Hybrid Network Intrusion Triage\n",
"\n",
"\n",
"**Topics:** classifier, anomaly score, hybrid policy, alert budget, SOC report\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Compare three alert policies on the fixed NSL-KDD test set:\n",
"\n",
"1. a supervised Decision Tree for known attack patterns;\n",
"2. a normal-only Isolation Forest for anomaly hunting;\n",
"3. a hybrid OR policy that alerts when either detector fires.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/hybrid_ids_metrics.csv`\n",
"- `output/hybrid_ids_group_recall.csv`\n",
"- `output/hybrid_ids_top_alerts.csv`\n",
"- `output/hybrid_ids_confusion_matrix.png`\n",
"- `output/hybrid_ids_score_distribution.png`\n",
"- `output/hybrid_ids_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"TRAIN_PATH = Path(\"data/nsl_kdd_train.csv\")\n",
"TEST_PATH = Path(\"data/nsl_kdd_test.csv\")\n",
"\n",
"missing_files = [str(path) for path in (TRAIN_PATH, TEST_PATH) if not path.exists()]\n",
"if missing_files:\n",
" raise FileNotFoundError(\n",
" \"Missing prepared NSL-KDD files: \" + \", \".join(missing_files) +\n",
" \". Follow data/README.md and run: python data/prepare_dataset.py\"\n",
" )\n",
"\n",
"TARGET_COLUMNS = [\"attack_label\", \"attack_group\", \"binary_label\", \"difficulty\"]\n",
"CATEGORICAL_FEATURES = [\"protocol_type\", \"service\", \"flag\"]\n",
"\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import IsolationForest\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import ConfusionMatrixDisplay\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"\n",
"\n",
"from sklearn.metrics import (\n",
" accuracy_score, average_precision_score, confusion_matrix,\n",
" f1_score, precision_score, recall_score,\n",
")\n",
"\n",
"\n",
"def security_metrics(y_true, y_pred):\n",
" tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()\n",
" return {\n",
" \"accuracy\": accuracy_score(y_true, y_pred),\n",
" \"attack_precision\": precision_score(y_true, y_pred, zero_division=0),\n",
" \"attack_recall\": recall_score(y_true, y_pred, zero_division=0),\n",
" \"attack_f1\": f1_score(y_true, y_pred, zero_division=0),\n",
" \"false_positives\": int(fp),\n",
" \"false_negatives\": int(fn),\n",
" \"false_positive_rate\": fp / (fp + tn) if fp + tn else 0.0,\n",
" \"alerts_per_10000\": 10000 * (fp + tp) / len(y_true),\n",
" }\n",
"\n",
"\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-004",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: load the fixed split and build the supervised feature matrices.\n",
"train_df = None\n",
"test_df = None\n",
"feature_cols = None\n",
"numeric_features = None\n",
"X_train, y_train, X_test, y_test = None, None, None, None\n",
"\n",
"# TODO 2: fit the bounded, balanced Decision Tree pipeline from Lab 5.\n",
"tree_model = None\n",
"tree_predictions = None\n",
"tree_probabilities = None"
]
},
{
"cell_type": "markdown",
"id": "cell-005",
"metadata": {},
"source": [
"## 1. Fit a normal-only Isolation Forest and set its alert budget"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-006",
"metadata": {},
"outputs": [],
"source": [
"# TODO 3: define anomaly_features, split normal training data, fit an\n",
"# Isolation Forest pipeline, and select the 99th validation percentile.\n",
"anomaly_features = None\n",
"normal_fit = None\n",
"normal_valid = None\n",
"anomaly_model = None\n",
"validation_scores = None\n",
"anomaly_scores = None\n",
"anomaly_threshold = None\n",
"anomaly_predictions = None"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Compare classifier, anomaly, and hybrid policies"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"# TODO 4: create Classifier, Anomaly, and Hybrid_OR predictions/scores.\n",
"policy_predictions = None\n",
"policy_scores = None\n",
"\n",
"# TODO 5: build metrics_table and select the highest-recall policy with\n",
"# FPR <= 0.10, breaking ties by F1 then lower FPR.\n",
"metrics_table = None\n",
"selected_policy = None\n",
"selected_predictions = None\n",
"display(metrics_table)"
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Audit attack groups and create the alert evidence table"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"# TODO 6: calculate per-group recall for every policy.\n",
"group_recall = None\n",
"\n",
"# TODO 7: build alert_table, derive alert_reason, and keep the top 200\n",
"# selected alerts ordered by anomaly score and tree probability.\n",
"alert_table = None\n",
"top_alerts = None\n",
"display(top_alerts)"
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Export evidence and write the SOC recommendation"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: save a confusion matrix and anomaly-score distribution chart.\n",
"\n",
"# TODO 9: write an English SOC summary covering the selected policy,\n",
"# metrics, alert volume, detector roles, weak groups, and limitations.\n",
"analyst_summary = None\n",
"\n",
"# TODO 10: export the metrics, group recall, top alerts, and summary.\n",
"print(analyst_summary)"
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"hybrid_ids_metrics.csv\", \"hybrid_ids_group_recall.csv\",\n",
" \"hybrid_ids_top_alerts.csv\", \"hybrid_ids_confusion_matrix.png\",\n",
" \"hybrid_ids_score_distribution.png\", \"hybrid_ids_analyst_summary.md\",\n",
"}\n",
"assert set(policy_predictions) == {\"Classifier\", \"Anomaly\", \"Hybrid_OR\"}\n",
"assert selected_policy in policy_predictions\n",
"assert len(alert_table) == len(test_df)\n",
"assert set(group_recall[\"attack_group\"]) == {\"DoS\", \"Probe\", \"R2L\", \"U2R\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"assert \"NSL-KDD\" in analyst_summary\n",
"print(\"✓ Lab 6 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom.\n",
"2. Confirm that all six output files exist.\n",
"3. Verify that the selected policy follows the documented rule.\n",
"4. Explain how anomaly-only alerts differ from classifier-only alerts.\n",
"5. Identify the weakest attack group and one dataset limitation."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/07_Botnet_Flow_Detection_CTU13_Lab.ipynb
+356
View File
@@ -0,0 +1,356 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cell-001",
"metadata": {},
"source": [
"# Module 6 — Lab 7: Botnet Flow Detection with CTU-13\n",
"\n",
"\n",
"**Topics:** CTU-13, flow labels, botnet classification, threshold selection, source triage\n",
"\n",
"> **Lab version:** Complete every `TODO`, run the cells in order, and use the self-check before opening the Solution."
]
},
{
"cell_type": "markdown",
"id": "cell-002",
"metadata": {},
"source": [
"## Mission\n",
"\n",
"Build a lightweight botnet detector from one labeled CTU-13 `.binetflow` scenario.\n",
"\n",
"The workflow deliberately separates the three CTU-13 label families:\n",
"\n",
"- **Botnet:** verified traffic produced by an infected host;\n",
"- **Normal:** verified traffic produced by known normal hosts;\n",
"- **Background:** traffic whose status is not known with confidence.\n",
"\n",
"Train and evaluate only on **Botnet versus Normal** records. Score Background flows for analyst triage, but do not treat them as verified benign ground truth.\n",
"\n",
"**Required outputs**\n",
"\n",
"- `output/ctu13_botnet_metrics.csv`\n",
"- `output/ctu13_source_triage.csv`\n",
"- `output/ctu13_top_flows.csv`\n",
"- `output/ctu13_confusion_matrix.png`\n",
"- `output/ctu13_score_distribution.png`\n",
"- `output/ctu13_analyst_summary.md`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-003",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DATA_PATH = Path(\"data/ctu13_botnet_flows.csv\")\n",
"OUTPUT_DIR = Path(\"output\")\n",
"OUTPUT_DIR.mkdir(exist_ok=True)\n",
"\n",
"if not DATA_PATH.exists():\n",
" raise FileNotFoundError(\n",
" f\"Missing {DATA_PATH}. Follow data/README.md and run \"\n",
" \"python data/prepare_ctu13.py <path-to-scenario.binetflow>\"\n",
" )\n",
"\n",
"from sklearn.compose import ColumnTransformer\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.metrics import (\n",
" ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n",
" confusion_matrix, f1_score, precision_score, recall_score,\n",
")\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"\n",
"RANDOM_STATE = 42"
]
},
{
"cell_type": "markdown",
"id": "cell-004",
"metadata": {},
"source": [
"## 1. Audit the prepared scenario\n",
"\n",
"Addresses are valuable evidence for source-level triage, but they must not become model features. A model that memorizes a scenario's infected IP address would appear accurate without learning transferable network behavior."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-005",
"metadata": {},
"outputs": [],
"source": [
"# TODO 1: Load DATA_PATH into flows and inspect its shape and label counts.\n",
"flows = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-006",
"metadata": {},
"source": [
"### Audit questions\n",
"\n",
"1. Why is Background not equivalent to Normal?\n",
"2. Why would `src_addr`, `dst_addr`, `raw_label`, or `label_family` cause leakage if used as model inputs?\n",
"3. Does this prepared sample contain all three label families?"
]
},
{
"cell_type": "markdown",
"id": "cell-007",
"metadata": {},
"source": [
"## 2. Define behavioral features and create three data splits\n",
"\n",
"Use flow measurements and low-cardinality connection metadata. Keep addresses only as investigation evidence.\n",
"\n",
"The validation set selects an operating threshold. The test set remains untouched until the final evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-008",
"metadata": {},
"outputs": [],
"source": [
"categorical_features = [\"protocol\", \"direction\", \"state\"]\n",
"numeric_features = [\n",
" \"duration\", \"src_port\", \"dst_port\", \"src_tos\", \"dst_tos\",\n",
" \"total_packets\", \"total_bytes\", \"src_bytes\", \"dst_bytes\",\n",
" \"bytes_per_packet\", \"src_byte_ratio\", \"start_hour\",\n",
" \"is_well_known_dst_port\",\n",
"]\n",
"feature_columns = categorical_features + numeric_features\n",
"\n",
"\n",
"# TODO 2: Keep only Botnet and Normal rows in `labeled`.\n",
"# TODO 3: Put Background rows in a separate `background` table.\n",
"labeled = ...\n",
"background = ...\n",
"X = ...\n",
"y = ...\n",
"\n",
"# TODO 4: Create stratified 60% train, 20% validation, and 20% test splits.\n",
"X_train_valid, X_test, y_train_valid, y_test = ...\n",
"X_train, X_valid, y_train, y_valid = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-009",
"metadata": {},
"source": [
"## 3. Train a bounded Random Forest\n",
"\n",
"The pipeline imputes missing values and one-hot encodes categorical fields. The forest is intentionally bounded to keep runtime and model complexity suitable for a classroom computer."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-010",
"metadata": {},
"outputs": [],
"source": [
"preprocessor = ColumnTransformer([\n",
" (\"categorical\", Pipeline([\n",
" (\"imputer\", SimpleImputer(strategy=\"most_frequent\")),\n",
" (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
" ]), categorical_features),\n",
" (\"numeric\", SimpleImputer(strategy=\"median\"), numeric_features),\n",
"])\n",
"\n",
"detector = Pipeline([\n",
" (\"preprocessor\", preprocessor),\n",
" (\"model\", RandomForestClassifier(\n",
" n_estimators=200,\n",
" max_depth=12,\n",
" min_samples_leaf=3,\n",
" class_weight=\"balanced_subsample\",\n",
" random_state=RANDOM_STATE,\n",
" n_jobs=-1,\n",
" )),\n",
"])\n",
"\n",
"\n",
"# TODO 5: Fit the pipeline on the training split.\n",
"# TODO 6: Calculate Botnet probabilities for the validation split.\n",
"...\n",
"validation_scores = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-011",
"metadata": {},
"source": [
"## 4. Select a validation threshold under an alert-quality constraint\n",
"\n",
"Evaluate thresholds from 0.05 to 0.95. Among thresholds with a validation false-positive rate no greater than 5%, select the one with the highest Botnet recall, then F1. If no threshold satisfies the constraint, select the highest-F1 threshold."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-012",
"metadata": {},
"outputs": [],
"source": [
"# TODO 7: Build threshold_table and select operating_threshold using the rule above.\n",
"threshold_table = ...\n",
"operating_threshold = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-013",
"metadata": {},
"source": [
"## 5. Evaluate once on the held-out test set\n",
"\n",
"Report Botnet precision, recall, F1, PR-AUC, false-positive rate, and alerts per 10,000 labeled flows. Accuracy is included only for context."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-014",
"metadata": {},
"outputs": [],
"source": [
"# TODO 8: Score the test split, apply operating_threshold, and create metrics_table.\n",
"test_scores = ...\n",
"test_predictions = ...\n",
"metrics_table = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-015",
"metadata": {},
"source": [
"## 6. Score all flows and create source-level triage\n",
"\n",
"Apply the detector to Botnet, Normal, and Background records. Aggregation by source address is an analyst view, not a model feature.\n",
"\n",
"A high Background score is a lead for investigation, not proof that the flow is malicious."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-016",
"metadata": {},
"outputs": [],
"source": [
"# TODO 9: Score every flow and add botnet_score and botnet_alert columns.\n",
"# TODO 10: Aggregate flow count, score statistics, and alert count by src_addr.\n",
"all_scores = ...\n",
"source_triage = ...\n",
"top_flows = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-017",
"metadata": {},
"source": [
"## 7. Export evidence and write the analyst summary\n",
"\n",
"The summary must distinguish measured performance on verified labels from exploratory scoring of uncertain Background traffic."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-018",
"metadata": {},
"outputs": [],
"source": [
"# TODO 11: Export the three CSV files, two figures, and analyst summary.\n",
"analyst_summary = ..."
]
},
{
"cell_type": "markdown",
"id": "cell-019",
"metadata": {},
"source": [
"## Self-check"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cell-020",
"metadata": {},
"outputs": [],
"source": [
"expected_files = {\n",
" \"ctu13_botnet_metrics.csv\", \"ctu13_source_triage.csv\", \"ctu13_top_flows.csv\",\n",
" \"ctu13_confusion_matrix.png\", \"ctu13_score_distribution.png\",\n",
" \"ctu13_analyst_summary.md\",\n",
"}\n",
"assert set(flows[\"label_family\"]).issubset({\"Botnet\", \"Normal\", \"Background\"})\n",
"assert not {\"src_addr\", \"dst_addr\", \"raw_label\", \"label_family\", \"binary_label\"}.intersection(feature_columns)\n",
"assert set(y_test.unique()) == {0, 1}\n",
"assert 0.0 <= operating_threshold <= 1.0\n",
"assert len(test_scores) == len(y_test)\n",
"assert set(source_triage.columns) >= {\"src_addr\", \"flow_count\", \"botnet_alerts\", \"alert_rate\"}\n",
"assert expected_files.issubset({path.name for path in OUTPUT_DIR.iterdir()})\n",
"print(\"✓ Lab 7 completed\")"
]
},
{
"cell_type": "markdown",
"id": "cell-021",
"metadata": {},
"source": [
"## Submission checklist\n",
"\n",
"1. Run the notebook from top to bottom and pass the self-check.\n",
"2. Submit the completed Lab notebook and six generated output files.\n",
"3. Explain why Background was excluded from supervised metrics.\n",
"4. Describe why IP addresses are evidence fields rather than model features.\n",
"5. Identify the highest-priority source and cite at least two supporting flow indicators.\n",
"6. Propose a cross-scenario test that would provide stronger evidence of generalization."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Lab-6/Lab-6-Practical/data/README.md
+174
View File
@@ -0,0 +1,174 @@
# Dataset setup — NSL-KDD
All six notebooks use the fixed NSL-KDD train/test split. NSL-KDD retains the 41 KDD network features while reducing the duplicate-record problem of the original KDD Cup 1999 release.
## Download
The official [UNB NSL-KDD page](https://www.unb.ca/cic/datasets/nsl.html) documents the dataset, citation, and redistribution conditions, but it no longer provides direct file downloads.
Download these two files from the [NSL-KDD GitHub mirror](https://github.com/defcom17/NSL_KDD):
```text
KDDTrain+_20Percent.txt
KDDTest+.txt
```
A preserved copy is also available through the [NSL-KDD Zenodo record](https://zenodo.org/records/17424143). If the Zenodo record does not include the 20% training file, download `KDDTrain+.txt`; the preparation command accepts it when `--full-train` is supplied.
Place the files as follows:
```text
Practical-6/
└── data/
├── prepare_dataset.py
└── raw/
├── KDDTrain+_20Percent.txt
└── KDDTest+.txt
```
## Prepare the classroom tables
From `Practical-6`, run:
```bash
python data/prepare_dataset.py
```
To use the full training set instead, place `KDDTrain+.txt` under `data/raw/` and run:
```bash
python data/prepare_dataset.py --full-train
```
The utility:
- applies the official 41 feature names;
- adds `attack_label`, `attack_group`, `binary_label`, and `difficulty`;
- validates numeric and categorical columns;
- rejects unknown attack labels so mapping errors remain visible;
- preserves the original fixed train/test split;
- writes `nsl_kdd_train.csv`, `nsl_kdd_test.csv`, and `feature_manifest.csv`.
## Quick verification
```python
from pathlib import Path
import pandas as pd
train = pd.read_csv(Path("data/nsl_kdd_train.csv"))
test = pd.read_csv(Path("data/nsl_kdd_test.csv"))
print("Train:", train.shape)
print("Test:", test.shape)
print(train["binary_label"].value_counts().sort_index())
print(test["attack_group"].value_counts())
```
Each prepared table must have 41 model features plus these four audit/target columns:
```text
attack_label, attack_group, binary_label, difficulty
```
## Citation and redistribution
The official UNB page permits redistribution but requires citation of NSL-KDD and the associated paper:
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” CISDA, 2009.
Retain this citation when redistributing prepared classroom data.
## Important limitations
- NSL-KDD is derived from traffic collected for the KDD Cup 1999 benchmark.
- The fixed test set contains attack types not present in the training set; this supports a useful generalization discussion but does not make the benchmark modern.
- Class prevalence does not represent a production SOC.
- IP addresses, packet payloads, timestamps, and modern encrypted-flow characteristics are absent.
- The dataset has no dedicated botnet label.
---
# Dataset setup — CTU-13 for Lab 7
CTU-13 contains 13 real-network botnet scenarios with flow-level Botnet, Normal, and Background labels. Lab 7 uses the labeled bidirectional `.binetflow` file from one scenario; PCAP files and executable samples are not required.
## Recommended download
Use the official [CTU-13 dataset page](https://www.stratosphereips.org/datasets-ctu13) and download Scenario 8, listed as `CTU-Malware-Capture-Botnet-49`. Locate this file inside the scenario's `detailed-bidirectional-flow-labels` directory:
```text
capture20110816-3.binetflow
```
Scenario 8 is recommended because it matches the botnet detection reference used for this practical. Another labeled CTU-13 scenario can be used, but class counts and model results will differ.
The official page also provides the complete `CTU-13-Dataset.tar.bz2` archive. The compressed archive is approximately 1.9 GB, so downloading one scenario is usually more convenient for a classroom exercise.
## Placement and preparation
Place the `.binetflow` file below `data/raw/`:
```text
Practical-6/
└── data/
├── prepare_ctu13.py
└── raw/
└── capture20110816-3.binetflow
```
From `Practical-6`, run:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow
```
If exactly one `.binetflow` file exists below `data/raw/`, the input argument may be omitted:
```bash
python data/prepare_ctu13.py
```
The utility reads the source in chunks and retains at most 20,000 records from each label family. To adjust the classroom sample size or memory usage:
```bash
python data/prepare_ctu13.py data/raw/capture20110816-3.binetflow --per-class 30000 --chunk-size 100000
```
It creates:
```text
data/ctu13_botnet_flows.csv
```
The loader accepts both common CTU-13 flow schemas:
- the compact 15-column export containing `Dir`;
- the extended Argus export containing directional counters such as
`SrcPkts`, `DstPkts`, and `DstBytes` but no `Dir`.
For the extended schema, the classroom `direction` feature is derived as
`bidirectional`, `source_only`, `destination_only`, or `unknown` from the source
and destination packet counters. When available, the original `DstBytes` value
is used instead of reconstructing it from total and source bytes.
## Label policy
- `Botnet` receives `binary_label = 1`.
- `Normal` receives `binary_label = 0`.
- `Background` receives `binary_label = -1` because its status is uncertain.
Lab 7 trains and evaluates on Botnet versus Normal only. Background flows are scored for investigation but excluded from supervised performance metrics.
## Safety and leakage controls
- Use only `.binetflow` flow records; do not download or execute malware samples.
- Source and destination IP addresses are retained for analyst triage but excluded from model features.
- Raw label fields are excluded from model features.
- The preparation utility applies bounded sampling so the notebook does not need to load the complete raw scenario.
- A same-scenario random split is suitable for instruction but does not demonstrate cross-botnet generalization.
## Citation
S. Garcia, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” *Computers & Security*, vol. 45, pp. 100–123, 2014. https://doi.org/10.1016/j.cose.2014.05.011
CTU-13 is distributed under the Creative Commons Attribution license. Preserve the dataset citation and attribution in redistributed course materials.
Lab-6/Lab-6-Practical/data/ctu13_botnet_flows.csv
+46128
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/feature_manifest.csv
+42
View File
@@ -0,0 +1,42 @@
feature,feature_type,model_feature
duration,numeric,True
protocol_type,categorical,True
service,categorical,True
flag,categorical,True
src_bytes,numeric,True
dst_bytes,numeric,True
land,numeric,True
wrong_fragment,numeric,True
urgent,numeric,True
hot,numeric,True
num_failed_logins,numeric,True
logged_in,numeric,True
num_compromised,numeric,True
root_shell,numeric,True
su_attempted,numeric,True
num_root,numeric,True
num_file_creations,numeric,True
num_shells,numeric,True
num_access_files,numeric,True
num_outbound_cmds,numeric,True
is_host_login,numeric,True
is_guest_login,numeric,True
count,numeric,True
srv_count,numeric,True
serror_rate,numeric,True
srv_serror_rate,numeric,True
rerror_rate,numeric,True
srv_rerror_rate,numeric,True
same_srv_rate,numeric,True
diff_srv_rate,numeric,True
srv_diff_host_rate,numeric,True
dst_host_count,numeric,True
dst_host_srv_count,numeric,True
dst_host_same_srv_rate,numeric,True
dst_host_diff_srv_rate,numeric,True
dst_host_same_src_port_rate,numeric,True
dst_host_srv_diff_host_rate,numeric,True
dst_host_serror_rate,numeric,True
dst_host_srv_serror_rate,numeric,True
dst_host_rerror_rate,numeric,True
dst_host_srv_rerror_rate,numeric,True
1 feature feature_type model_feature
2 duration numeric True
3 protocol_type categorical True
4 service categorical True
5 flag categorical True
6 src_bytes numeric True
7 dst_bytes numeric True
8 land numeric True
9 wrong_fragment numeric True
10 urgent numeric True
11 hot numeric True
12 num_failed_logins numeric True
13 logged_in numeric True
14 num_compromised numeric True
15 root_shell numeric True
16 su_attempted numeric True
17 num_root numeric True
18 num_file_creations numeric True
19 num_shells numeric True
20 num_access_files numeric True
21 num_outbound_cmds numeric True
22 is_host_login numeric True
23 is_guest_login numeric True
24 count numeric True
25 srv_count numeric True
26 serror_rate numeric True
27 srv_serror_rate numeric True
28 rerror_rate numeric True
29 srv_rerror_rate numeric True
30 same_srv_rate numeric True
31 diff_srv_rate numeric True
32 srv_diff_host_rate numeric True
33 dst_host_count numeric True
34 dst_host_srv_count numeric True
35 dst_host_same_srv_rate numeric True
36 dst_host_diff_srv_rate numeric True
37 dst_host_same_src_port_rate numeric True
38 dst_host_srv_diff_host_rate numeric True
39 dst_host_serror_rate numeric True
40 dst_host_srv_serror_rate numeric True
41 dst_host_rerror_rate numeric True
42 dst_host_srv_rerror_rate numeric True
Lab-6/Lab-6-Practical/data/nsl_kdd_test.csv
+22545
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/nsl_kdd_train.csv
+125974
View File
File diff suppressed because it is too large Load Diff
Lab-6/Lab-6-Practical/data/prepare_ctu13.py
+261
View File
@@ -0,0 +1,261 @@
"""Create a bounded, classroom-ready flow table from one CTU-13 .binetflow file."""
from __future__ import annotations
import argparse
from pathlib import Path
import numpy as np
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
REQUIRED_RAW_COLUMNS = [
"StartTime", "Dur", "Proto", "SrcAddr", "Sport", "DstAddr", "Dport",
"State", "sTos", "dTos", "TotPkts", "TotBytes", "SrcBytes", "Label",
]
# CTU-13 is distributed in more than one Argus text schema. The compact
# 15-column export contains ``Dir``. The extended export supplied with some
# scenarios omits it but provides per-direction packet and byte counters.
OPTIONAL_RAW_COLUMNS = ["Dir", "SrcPkts", "DstPkts", "DstBytes"]
RENAME_MAP = {
"StartTime": "start_time",
"Dur": "duration",
"Proto": "protocol",
"SrcAddr": "src_addr",
"Sport": "src_port",
"Dir": "direction",
"DstAddr": "dst_addr",
"Dport": "dst_port",
"State": "state",
"sTos": "src_tos",
"dTos": "dst_tos",
"TotPkts": "total_packets",
"TotBytes": "total_bytes",
"SrcBytes": "src_bytes",
"DstBytes": "dst_bytes",
"Label": "raw_label",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"input",
nargs="?",
type=Path,
help="Path to one labeled CTU-13 .binetflow file. If omitted, data/raw is searched.",
)
parser.add_argument(
"--per-class",
type=int,
default=20_000,
help="Maximum sampled rows for each of Botnet, Normal, and Background.",
)
parser.add_argument(
"--chunk-size",
type=int,
default=200_000,
help="Rows read at a time; reduce this value on low-memory computers.",
)
parser.add_argument(
"--output",
type=Path,
default=HERE / "ctu13_botnet_flows.csv",
help="Prepared CSV destination.",
)
return parser.parse_args()
def find_input(requested: Path | None) -> Path:
if requested is not None:
path = requested.expanduser().resolve()
if not path.is_file():
raise FileNotFoundError(f"CTU-13 input file not found: {path}")
return path
matches = sorted(RAW_DIR.rglob("*.binetflow")) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one .binetflow file below {RAW_DIR}, found {len(matches)}. "
"Pass the input path explicitly when more than one file is present."
)
return matches[0]
def label_family(labels: pd.Series) -> pd.Series:
text = labels.fillna("").astype(str)
return pd.Series(
np.select(
[
text.str.contains("Botnet", case=False, regex=False),
text.str.contains("Normal", case=False, regex=False),
text.str.contains("Background", case=False, regex=False),
],
["Botnet", "Normal", "Background"],
default="Other",
),
index=labels.index,
dtype="string",
)
def parse_port(values: pd.Series) -> pd.Series:
text = values.fillna("").astype(str).str.strip().str.lower()
decimal = pd.to_numeric(text.where(~text.str.startswith("0x")), errors="coerce")
hexadecimal = text.where(text.str.startswith("0x")).map(
lambda value: int(value, 16) if isinstance(value, str) and value else np.nan
)
return decimal.fillna(hexadecimal).astype("Float64")
def inspect_schema(path: Path) -> tuple[list[str], str]:
"""Validate a compact or extended CTU-13 header and select needed fields."""
header = pd.read_csv(path, nrows=0).columns.tolist()
missing = sorted(set(REQUIRED_RAW_COLUMNS) - set(header))
if missing:
raise ValueError(
f"Missing required CTU-13 columns: {missing}. Available columns: {header}"
)
selected = REQUIRED_RAW_COLUMNS + [
column for column in OPTIONAL_RAW_COLUMNS if column in header
]
if "Dir" in header:
direction_mode = "source Dir column"
elif {"SrcPkts", "DstPkts"}.issubset(header):
direction_mode = "derived from SrcPkts/DstPkts"
else:
direction_mode = "constant unknown (no Dir or packet-direction counters)"
print(f"Detected {len(header)}-column CTU-13 schema; direction is {direction_mode}.")
return selected, direction_mode
def ensure_direction(frame: pd.DataFrame) -> pd.DataFrame:
"""Return a frame with a categorical Dir field for either CTU-13 schema."""
if "Dir" in frame.columns:
frame["Dir"] = frame["Dir"].fillna("unknown").astype(str).str.strip()
frame.loc[frame["Dir"].eq(""), "Dir"] = "unknown"
return frame
if {"SrcPkts", "DstPkts"}.issubset(frame.columns):
src_packets = pd.to_numeric(frame["SrcPkts"], errors="coerce").fillna(0)
dst_packets = pd.to_numeric(frame["DstPkts"], errors="coerce").fillna(0)
frame["Dir"] = np.select(
[
(src_packets > 0) & (dst_packets > 0),
(src_packets > 0) & (dst_packets <= 0),
(src_packets <= 0) & (dst_packets > 0),
],
["bidirectional", "source_only", "destination_only"],
default="unknown",
)
else:
frame["Dir"] = "unknown"
return frame
def bounded_sample(path: Path, per_class: int, chunk_size: int) -> pd.DataFrame:
if per_class < 1 or chunk_size < 1:
raise ValueError("--per-class and --chunk-size must be positive integers.")
kept = {family: pd.DataFrame() for family in ("Botnet", "Normal", "Background")}
rng = np.random.default_rng(42)
saw_header = False
selected_columns, _ = inspect_schema(path)
for chunk_number, chunk in enumerate(
pd.read_csv(
path,
usecols=selected_columns,
chunksize=chunk_size,
low_memory=False,
),
start=1,
):
saw_header = True
chunk = ensure_direction(chunk)
output_columns = REQUIRED_RAW_COLUMNS + ["Dir"]
if "DstBytes" in chunk.columns:
output_columns.append("DstBytes")
chunk = chunk[output_columns].copy()
chunk["label_family"] = label_family(chunk["Label"])
for family in kept:
part = chunk.loc[chunk["label_family"].eq(family)].copy()
if part.empty:
continue
part["_sample_key"] = rng.random(len(part))
candidates = pd.concat([kept[family], part], ignore_index=True)
kept[family] = candidates.nsmallest(per_class, "_sample_key")
counts = {family: len(frame) for family, frame in kept.items()}
print(f"Processed chunk {chunk_number}: retained {counts}")
if not saw_header:
raise ValueError(f"No rows found in {path}")
sampled = pd.concat(kept.values(), ignore_index=True).drop(columns="_sample_key")
if sampled.empty:
raise ValueError("No Botnet, Normal, or Background labels were found.")
return sampled
def engineer_features(sampled: pd.DataFrame) -> pd.DataFrame:
frame = sampled.rename(columns=RENAME_MAP).copy()
frame["start_time"] = pd.to_datetime(frame["start_time"], errors="coerce")
numeric = [
"duration", "src_tos", "dst_tos", "total_packets", "total_bytes", "src_bytes"
]
if "dst_bytes" in frame.columns:
numeric.append("dst_bytes")
for column in numeric:
frame[column] = pd.to_numeric(frame[column], errors="coerce")
frame["src_port"] = parse_port(frame["src_port"])
frame["dst_port"] = parse_port(frame["dst_port"])
derived_dst_bytes = (frame["total_bytes"] - frame["src_bytes"]).clip(lower=0)
if "dst_bytes" in frame.columns:
frame["dst_bytes"] = frame["dst_bytes"].fillna(derived_dst_bytes).clip(lower=0)
else:
frame["dst_bytes"] = derived_dst_bytes
frame["bytes_per_packet"] = frame["total_bytes"] / frame["total_packets"].clip(lower=1)
frame["src_byte_ratio"] = frame["src_bytes"] / frame["total_bytes"].clip(lower=1)
frame["start_hour"] = frame["start_time"].dt.hour
frame["is_well_known_dst_port"] = frame["dst_port"].between(0, 1023).astype("Int64")
frame["binary_label"] = frame["label_family"].map(
{"Normal": 0, "Botnet": 1, "Background": -1}
).astype(int)
ordered = [
"start_time", "duration", "protocol", "src_addr", "src_port", "direction",
"dst_addr", "dst_port", "state", "src_tos", "dst_tos", "total_packets",
"total_bytes", "src_bytes", "dst_bytes", "bytes_per_packet", "src_byte_ratio",
"start_hour", "is_well_known_dst_port", "raw_label", "label_family", "binary_label",
]
return frame[ordered].sort_values("start_time", na_position="last").reset_index(drop=True)
def main() -> None:
args = parse_args()
input_path = find_input(args.input)
output_path = args.output.expanduser().resolve()
output_path.parent.mkdir(parents=True, exist_ok=True)
sampled = bounded_sample(input_path, args.per_class, args.chunk_size)
prepared = engineer_features(sampled)
prepared.to_csv(output_path, index=False)
print(f"Created {output_path} with shape {prepared.shape}")
print(prepared["label_family"].value_counts())
print("Background is retained for triage but excluded from supervised ground-truth metrics.")
if __name__ == "__main__":
main()
Lab-6/Lab-6-Practical/data/prepare_dataset.py
+123
View File
@@ -0,0 +1,123 @@
"""Prepare fixed NSL-KDD train/test tables for Module 6 notebooks."""
from __future__ import annotations
import argparse
from pathlib import Path
import pandas as pd
HERE = Path(__file__).resolve().parent
RAW_DIR = HERE / "raw"
FEATURE_NAMES = [
"duration", "protocol_type", "service", "flag", "src_bytes", "dst_bytes",
"land", "wrong_fragment", "urgent", "hot", "num_failed_logins",
"logged_in", "num_compromised", "root_shell", "su_attempted", "num_root",
"num_file_creations", "num_shells", "num_access_files", "num_outbound_cmds",
"is_host_login", "is_guest_login", "count", "srv_count", "serror_rate",
"srv_serror_rate", "rerror_rate", "srv_rerror_rate", "same_srv_rate",
"diff_srv_rate", "srv_diff_host_rate", "dst_host_count",
"dst_host_srv_count", "dst_host_same_srv_rate", "dst_host_diff_srv_rate",
"dst_host_same_src_port_rate", "dst_host_srv_diff_host_rate",
"dst_host_serror_rate", "dst_host_srv_serror_rate", "dst_host_rerror_rate",
"dst_host_srv_rerror_rate",
]
CATEGORICAL_FEATURES = ["protocol_type", "service", "flag"]
ATTACK_GROUPS = {
"normal": "Normal",
"back": "DoS", "land": "DoS", "neptune": "DoS", "pod": "DoS",
"smurf": "DoS", "teardrop": "DoS", "apache2": "DoS", "mailbomb": "DoS",
"processtable": "DoS", "udpstorm": "DoS", "worm": "DoS",
"ipsweep": "Probe", "nmap": "Probe", "portsweep": "Probe", "satan": "Probe",
"mscan": "Probe", "saint": "Probe",
"ftp_write": "R2L", "guess_passwd": "R2L", "imap": "R2L",
"multihop": "R2L", "phf": "R2L", "spy": "R2L", "warezclient": "R2L",
"warezmaster": "R2L", "sendmail": "R2L", "named": "R2L",
"snmpgetattack": "R2L", "snmpguess": "R2L", "xlock": "R2L",
"xsnoop": "R2L", "httptunnel": "R2L",
"buffer_overflow": "U2R", "loadmodule": "U2R", "perl": "U2R",
"rootkit": "U2R", "ps": "U2R", "sqlattack": "U2R", "xterm": "U2R",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--full-train",
action="store_true",
help="Use KDDTrain+.txt instead of KDDTrain+_20Percent.txt.",
)
return parser.parse_args()
def find_one(filename: str) -> Path:
matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
if len(matches) != 1:
raise FileNotFoundError(
f"Expected exactly one {filename!r} below {RAW_DIR}. "
"Follow data/README.md and verify the extracted files."
)
return matches[0]
def load_nsl_kdd(path: Path) -> pd.DataFrame:
columns = FEATURE_NAMES + ["attack_label", "difficulty"]
frame = pd.read_csv(path, header=None, names=columns, low_memory=False)
frame["attack_label"] = (
frame["attack_label"].astype(str).str.strip().str.rstrip(".").str.lower()
)
unknown = sorted(set(frame["attack_label"]) - set(ATTACK_GROUPS))
if unknown:
raise ValueError(f"Unknown attack labels in {path.name}: {unknown}")
for column in CATEGORICAL_FEATURES:
frame[column] = frame[column].astype(str).str.strip().str.lower()
numeric_features = [name for name in FEATURE_NAMES if name not in CATEGORICAL_FEATURES]
for column in numeric_features + ["difficulty"]:
frame[column] = pd.to_numeric(frame[column], errors="raise")
frame["attack_group"] = frame["attack_label"].map(ATTACK_GROUPS)
frame["binary_label"] = (frame["attack_group"] != "Normal").astype(int)
return frame[FEATURE_NAMES + ["attack_label", "attack_group", "binary_label", "difficulty"]]
def main() -> None:
args = parse_args()
train_name = "KDDTrain+.txt" if args.full_train else "KDDTrain+_20Percent.txt"
train = load_nsl_kdd(find_one(train_name))
test = load_nsl_kdd(find_one("KDDTest+.txt"))
train_path = HERE / "nsl_kdd_train.csv"
test_path = HERE / "nsl_kdd_test.csv"
manifest_path = HERE / "feature_manifest.csv"
train.to_csv(train_path, index=False)
test.to_csv(test_path, index=False)
manifest = pd.DataFrame({
"feature": FEATURE_NAMES,
"feature_type": [
"categorical" if name in CATEGORICAL_FEATURES else "numeric"
for name in FEATURE_NAMES
],
"model_feature": True,
})
manifest.to_csv(manifest_path, index=False)
print(f"Created {train_path} with shape {train.shape}")
print(f"Created {test_path} with shape {test.shape}")
print(f"Created {manifest_path}")
print("Train attack groups:")
print(train["attack_group"].value_counts())
print("Test attack groups:")
print(test["attack_group"].value_counts())
if __name__ == "__main__":
main()

Some files were not shown because too many files have changed in this diff Show More