labs/Lab-2/Lab-2-Practical/05_Pandas_Log_Cleaning_Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e51dfbb6",
   "metadata": {},
   "source": [
    "# Module 2 — Lab 5: Pandas — Loading, Inspecting, and Cleaning Logs\n",
    "\n",
    "\n",
    "**Topics:** CSV, DataFrame, missing values, duplicates, datetime, filtering\n",
    "\n",
    "\n",
    "> **Instructions:** Run the cells in order. Complete every `TODO`, then run the **Self-check** cell. Do not open the Solution notebook before attempting the tasks yourself.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e505f19b",
   "metadata": {},
   "source": [
    "## Scenario\n",
    "\n",
    "The `data/auth_log.csv` file contains duplicate rows, invalid timestamps, missing critical fields, and inconsistent capitalization. The goal is to produce a clean DataFrame ready for analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4a8981a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "\n",
    "DATA_PATH = Path(\"data/auth_log.csv\")\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "log_df = pd.read_csv(DATA_PATH)\n",
    "log_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aa72689",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "source": [
    "## 1. Inspect data quality\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bcc9310",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO 1: print the shape, duplicate count, missing-value counts, and dtypes\n",
    "print(\"Shape:\", None)\n",
    "print(\"Duplicate rows:\", None)\n",
    "print(\"Missing values by column:\")\n",
    "print(None)\n",
    "print(\"\\nData types:\")\n",
    "print(None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c13cc30",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "source": [
    "## 2. Remove duplicates and normalize timestamps and critical fields\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dc3dbd3",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "clean_df = log_df.copy()\n",
    "\n",
    "# TODO 2: remove duplicate rows\n",
    "\n",
    "# TODO 3: parse timestamp with pd.to_datetime using errors=\"coerce\" and utc=True\n",
    "\n",
    "# TODO 4: drop rows missing timestamp, source_ip, or status\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d39a43",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "source": [
    "## 3. Normalize text and create time-based fields\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e434ec2",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO 5: strip surrounding whitespace from source_ip\n",
    "# TODO 6: fill missing usernames with \"unknown\", strip whitespace, and convert to lowercase\n",
    "# TODO 7: strip whitespace and convert status to uppercase\n",
    "\n",
    "# TODO 8: retain only SUCCESS and FAILED status values\n",
    "# TODO 9: create the hour column from timestamp\n",
    "# TODO 10: sort by timestamp and reset the index\n",
    "\n",
    "clean_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94783144",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "source": [
    "## 4. Filter and aggregate\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef6981bc",
   "metadata": {
    "tags": [
     "exercise"
    ]
   },
   "outputs": [],
   "source": [
    "# TODO 11: filter FAILED rows\n",
    "failed_df = None\n",
    "\n",
    "# TODO 12: count FAILED events by source_ip\n",
    "failed_by_ip = None\n",
    "\n",
    "# TODO 13: count status values\n",
    "status_counts = None\n",
    "\n",
    "print(\"Status counts:\")\n",
    "print(status_counts)\n",
    "print(\"\\nFailed attempts by IP:\")\n",
    "print(failed_by_ip)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8ccd176",
   "metadata": {},
   "source": [
    "## 5. Export clean data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51b00290",
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df.to_csv(OUTPUT_DIR / \"clean_auth_log.csv\", index=False)\n",
    "clean_df.to_json(\n",
    "    OUTPUT_DIR / \"clean_auth_log.json\",\n",
    "    orient=\"records\",\n",
    "    indent=2,\n",
    "    date_format=\"iso\",\n",
    ")\n",
    "failed_by_ip.to_csv(\n",
    "    OUTPUT_DIR / \"failed_login_by_ip.csv\",\n",
    "    header=True,\n",
    ")\n",
    "\n",
    "print(\"Exported files to:\", OUTPUT_DIR.resolve())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "622a8262",
   "metadata": {},
   "source": [
    "## Self-check\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c5d54f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert log_df.duplicated().sum() >= 1\n",
    "assert clean_df.duplicated().sum() == 0\n",
    "assert clean_df[\"timestamp\"].isna().sum() == 0\n",
    "assert set(clean_df[\"status\"]) <= {\"SUCCESS\", \"FAILED\"}\n",
    "assert clean_df[\"username\"].str.strip().eq(clean_df[\"username\"]).all()\n",
    "assert failed_by_ip.index[0] == \"198.51.100.14\"\n",
    "assert (OUTPUT_DIR / \"clean_auth_log.csv\").exists()\n",
    "assert (OUTPUT_DIR / \"clean_auth_log.json\").exists()\n",
    "print(\"✓ Lab 5 completed\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}