CSAI2026 labs: Lab-2..Lab-6 (Practical + Solution, data), Module-2..Module-6 notebooks and datasets

2026-10-12 19:16:14 +07:00
commit 2fefa615cc
184 changed files with 1016761 additions and 0 deletions
@@ -0,0 +1,79 @@
+# Dataset setup — UCI Malware Static and Dynamic Features
+
+All six notebooks use pre-extracted tabular features from the official UCI dataset **Malware Static and Dynamic Features VxHeaven and Virus Total**.
+
+## Download
+
+1. Open the [UCI dataset page](https://archive.ics.uci.edu/dataset/541/malware%2Bstatic%2Band%2Bdynamic%2Bfeatures%2Bvxheaven%2Band%2Bvirus%2Btotal).
+2. Select **Download** and extract the archive.
+3. Create `data/raw/` if it does not already exist.
+4. Copy these three files into `data/raw/`:
+
+```text
+staDynBenignLab.csv
+staDynVxHeaven2698Lab.csv
+staDynVt2955Lab.csv
+```
+
+The expected layout is:
+
+```text
+Practical-5/
+└── data/
+    ├── prepare_dataset.py
+    └── raw/
+        ├── staDynBenignLab.csv
+        ├── staDynVxHeaven2698Lab.csv
+        └── staDynVt2955Lab.csv
+```
+
+## Prepare the classroom table
+
+From the `Practical-5` directory, run:
+
+```bash
+python data/prepare_dataset.py
+```
+
+The utility:
+
+- finds the three required files under `data/raw/`;
+- normalizes column names and adds deterministic occurrence suffixes when
+  case- or punctuation-distinct UCI fields collapse to the same snake-case name;
+- keeps columns shared by all three sources;
+- removes obvious identifiers and pre-existing target columns;
+- converts usable columns to numeric values;
+- adds `source` and `label` (`0 = benign`, `1 = malware`);
+- retains missing values so imputation can be fitted on training data inside the notebooks;
+- writes `data/malware_features.csv` and `data/feature_manifest.csv`.
+
+## Quick verification
+
+```python
+from pathlib import Path
+import pandas as pd
+
+path = Path("data/malware_features.csv")
+df = pd.read_csv(path)
+print(df.shape)
+print(df["label"].value_counts().sort_index())
+print(df["source"].value_counts())
+```
+
+The original release contains approximately 6,248 rows: 595 benign records and 5,653 malware records. The exact number of retained feature columns can vary if the source archive changes column formatting.
+
+## Provenance and license
+
+- Dataset: Malware Static and Dynamic Features VxHeaven and Virus Total
+- Source: UCI Machine Learning Repository
+- DOI: [10.24432/C58K6H](https://doi.org/10.24432/C58K6H)
+- License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
+
+Retain the citation and license when redistributing derived classroom tables.
+
+## Important limitations
+
+- The class distribution is strongly imbalanced.
+- Benign and malware samples originate from different collections, so source-specific artifacts may inflate random-split performance.
+- The features represent a historical dataset and should not be treated as a current production benchmark.
+- No raw malware is required or permitted for this practical.
@@ -0,0 +1,245 @@
+feature,non_missing_ratio,unique_values,feature_family
+add,1.0,2809,unclassified
+addressofentrypoint,1.0,3156,unclassified
+align,1.0,11,unclassified
+arg,1.0,5,unclassified
+asc,1.0,3,unclassified
+baseofcode,1.0,145,unclassified
+bool,1.0,2,unclassified
+bt,1.0,114,unclassified
+byte,1.0,17,unclassified
+call,1.0,2131,unclassified
+cexit,1.0,2,unclassified
+char,1.0,4,unclassified
+characteristics,1.0,52,unclassified
+checksum,1.0,2100,unclassified
+cld,1.0,685,unclassified
+cli,1.0,625,unclassified
+closehandle,1.0,2,unclassified
+cmc,1.0,635,unclassified
+cmp,1.0,1954,unclassified
+compile_date,1.0,3912,unclassified
+cookie,1.0,2,unclassified
+copyfilea,1.0,2,unclassified
+count_dll_loaded,1.0,60,unclassified
+count_file_copied,1.0,47,unclassified
+count_file_deleted,1.0,19,unclassified
+count_file_exists,1.0,75,unclassified
+count_file_opened,1.0,76,unclassified
+count_file_read,1.0,29,unclassified
+count_file_written,1.0,24,unclassified
+count_mutex,1.0,18,unclassified
+count_regkey_deleted,1.0,12,unclassified
+count_regkey_written,1.0,45,unclassified
+createdirectorya,1.0,2,unclassified
+createfilea,1.0,2,unclassified
+createmutexa,1.0,2,unclassified
+createprocessa,1.0,2,unclassified
+createthread,1.0,2,unclassified
+createtoolhelp32snapshot,1.0,2,unclassified
+cs,1.0,529,unclassified
+daa,1.0,618,unclassified
+data,1.0,4,unclassified
+data__2,1.0,2,unclassified
+db,1.0,1827,unclassified
+db_1,1.0,1147,unclassified
+dd,1.0,3111,unclassified
+dd_1,1.0,946,unclassified
+dec,1.0,1627,unclassified
+deletefilea,1.0,2,unclassified
+dll,1.0,225,unclassified
+dll_characteristics,1.0,27,unclassified
+ds,1.0,526,unclassified
+dw,1.0,63,unclassified
+enablewindow,1.0,2,unclassified
+ent_max,1.0,5674,unclassified
+ent_mean,1.0,6243,unclassified
+ent_median,1.0,6087,unclassified
+ent_min,1.0,5699,unclassified
+ent_var,1.0,6114,unclassified
+ent_whole_file,1.0,6248,unclassified
+entry,1.0,5,unclassified
+es,1.0,500,unclassified
+exit,1.0,33,unclassified
+exitprocess,1.0,3,unclassified
+exitwindowsex,1.0,2,unclassified
+faddp,1.0,131,unclassified
+fchs,1.0,55,unclassified
+fclose,1.0,2,unclassified
+fdiv,1.0,175,unclassified
+fdivp,1.0,130,unclassified
+fdivr,1.0,192,unclassified
+fild,1.0,276,unclassified
+file_alignment,1.0,5,unclassified
+files_operations,1.0,159,unclassified
+filesize,1.0,5252,unclassified
+findclose,1.0,2,unclassified
+findfirstfilea,1.0,2,unclassified
+findnextfilea,1.0,2,unclassified
+findwindowa,1.0,2,unclassified
+fistp,1.0,202,unclassified
+fld,1.0,144,unclassified
+fopen,1.0,3,unclassified
+free,1.0,169,unclassified
+fs,1.0,683,unclassified
+fstcw,1.0,19,unclassified
+fwrite,1.0,3,unclassified
+fxch,1.0,148,unclassified
+getcommandlinea,1.0,2,unclassified
+getfileattributesa,1.0,2,unclassified
+getforegroundwindow,1.0,2,unclassified
+getlasterror,1.0,4,unclassified
+getmainargs,1.0,2,unclassified
+getmodulefilenamea,1.0,3,unclassified
+getmodulehandlea,1.0,3,unclassified
+getstartupinfoa,1.0,2,unclassified
+getsystemdirectorya,1.0,2,unclassified
+getsystemmetrics,1.0,2,unclassified
+gettickcount,1.0,2,unclassified
+gs,1.0,796,unclassified
+handle,1.0,10,unclassified
+image_base,1.0,44,unclassified
+imul,1.0,1113,unclassified
+in,1.0,1016,unclassified
+inc,1.0,1941,unclassified
+initterm,1.0,2,unclassified
+int,1.0,1500,unclassified
+int_1,1.0,575,unclassified
+interlockedcompareexchange,1.0,2,unclassified
+interlockeddecrement,1.0,2,unclassified
+interlockedincrement,1.0,2,unclassified
+jb,1.0,1089,unclassified
+je,1.0,1732,unclassified
+jg,1.0,765,unclassified
+jge,1.0,748,unclassified
+jl,1.0,825,unclassified
+jmp,1.0,1707,unclassified
+jno,1.0,650,unclassified
+jo,1.0,819,unclassified
+lea,1.0,1673,unclassified
+loader_flags,1.0,4,unclassified
+loc,1.0,688,unclassified
+long,1.0,10,unclassified
+loope,1.0,580,unclassified
+lstrcata,1.0,2,unclassified
+lstrcpya,1.0,2,unclassified
+lstrlena,1.0,2,unclassified
+machine,1.0,2,unclassified
+magic,1.0,2,unclassified
+major_image_version,1.0,39,unclassified
+major_linker_version,1.0,30,unclassified
+major_operating_system_version,1.0,7,unclassified
+major_subsystem_version,1.0,5,unclassified
+malloc,1.0,5,unclassified
+max_min,1.0,5517,unclassified
+memcpy,1.0,5,unclassified
+memmove,1.0,3,unclassified
+memset,1.0,5,unclassified
+messageboxa,1.0,2,unclassified
+minor_image_version,1.0,44,unclassified
+minor_linker_version,1.0,47,unclassified
+minor_operating_system_version,1.0,10,unclassified
+minor_subsystem_version,1.0,6,unclassified
+mov,1.0,3156,unclassified
+mul,1.0,207,unclassified
+neg,1.0,301,unclassified
+nop,1.0,1037,unclassified
+not,1.0,265,unclassified
+number_of_iat_entires,1.0,467,unclassified
+number_of_iat_entires_1,1.0,467,unclassified
+number_of_imports_1,1.0,40,unclassified
+number_of_rva_and_sizes,1.0,8,unclassified
+number_of_rva_and_sizes_1,1.0,8,unclassified
+number_of_sections_1,1.0,29,unclassified
+number_of_symbols,1.0,50,unclassified
+off,1.0,4,unclassified
+or,1.0,1497,unclassified
+out,1.0,1013,unclassified
+p_fmode,1.0,2,unclassified
+pointer_to_symbol_table,1.0,42,unclassified
+pop,1.0,2306,unclassified
+popf,1.0,585,unclassified
+proc,1.0,2,unclassified
+process32first,1.0,2,unclassified
+process32next,1.0,2,unclassified
+ptr,1.0,16,unclassified
+push,1.0,3002,unclassified
+pushf,1.0,642,unclassified
+raiseexception,1.0,3,unclassified
+rand,1.0,6,unclassified
+rcl,1.0,314,unclassified
+rcr,1.0,261,unclassified
+rdata,1.0,2,unclassified
+rdtsc,1.0,50,unclassified
+regclosekey,1.0,3,unclassified
+regopenkeya,1.0,2,unclassified
+regopenkeyexa,1.0,2,unclassified
+regsetvalueexa,1.0,3,unclassified
+rep,1.0,263,unclassified
+ret,1.0,1458,unclassified
+rol,1.0,348,unclassified
+ror,1.0,332,unclassified
+sar,1.0,466,unclassified
+sbb,1.0,1230,unclassified
+scas,1.0,777,unclassified
+section_alignment,1.0,5,unclassified
+sendmessagea,1.0,2,unclassified
+set_app_type,1.0,3,unclassified
+setb,1.0,54,unclassified
+setcursorpos,1.0,2,unclassified
+setfileattributesa,1.0,3,unclassified
+setlasterror,1.0,2,unclassified
+setle,1.0,50,unclassified
+setusermatherr,1.0,2,unclassified
+setwindowtexta,1.0,2,unclassified
+shellexecutea,1.0,2,unclassified
+shl,1.0,493,unclassified
+shld,1.0,79,unclassified
+showwindow,1.0,2,unclassified
+shr,1.0,499,unclassified
+sidt,1.0,2,unclassified
+size_code,1.0,930,unclassified
+size_image,1.0,829,unclassified
+size_init_data,1.0,948,unclassified
+size_of_headers,1.0,30,unclassified
+size_of_heap_commit,1.0,10,unclassified
+size_of_heap_reserve,1.0,26,unclassified
+size_of_optional_header,1.0,2,unclassified
+size_of_stack_commit,1.0,35,unclassified
+size_of_stack_reserve,1.0,34,unclassified
+size_uninit_data,1.0,164,unclassified
+sizeofheaders,1.0,30,unclassified
+sizeofheaders_1,1.0,30,unclassified
+sleep,1.0,4,unclassified
+sprintf,1.0,2,unclassified
+ss,1.0,577,unclassified
+stc,1.0,634,unclassified
+std,1.0,658,unclassified
+sti,1.0,669,unclassified
+stos,1.0,790,unclassified
+strcat,1.0,4,unclassified
+strchr,1.0,2,unclassified
+strcmp,1.0,4,unclassified
+strcpy,1.0,4,unclassified
+strlen,1.0,6,unclassified
+strncmp,1.0,2,unclassified
+strstr,1.0,3,unclassified
+sub,1.0,1631,unclassified
+subsystem,1.0,3,unclassified
+test,1.0,1685,unclassified
+text,1.0,5,unclassified
+time,1.0,5,unclassified
+tlsalloc,1.0,2,unclassified
+tlsfree,1.0,2,unclassified
+tlsgetvalue,1.0,2,unclassified
+tlssetvalue,1.0,2,unclassified
+uint,1.0,3,unclassified
+unhandledexceptionfilter,1.0,2,unclassified
+waitforsingleobject,1.0,2,unclassified
+winexec,1.0,2,unclassified
+winmain,1.0,6,unclassified
+word,1.0,9,unclassified
+word__2,1.0,3,unclassified
+writefile,1.0,2,unclassified
+xchg,1.0,1217,unclassified
+xor,1.0,1887,unclassified
@@ -0,0 +1,168 @@
+"""Prepare a safe classroom feature table from the three UCI CSV files."""
+
+from pathlib import Path
+import re
+
+import numpy as np
+import pandas as pd
+
+
+HERE = Path(__file__).resolve().parent
+RAW_DIR = HERE / "raw"
+OUTPUT_PATH = HERE / "malware_features.csv"
+MANIFEST_PATH = HERE / "feature_manifest.csv"
+
+FILES = {
+    "benign_lab": ("staDynBenignLab.csv", 0),
+    "vxheaven": ("staDynVxHeaven2698Lab.csv", 1),
+    "virustotal": ("staDynVt2955Lab.csv", 1),
+}
+
+FORBIDDEN_NAMES = {
+    "label", "class", "target", "malware", "is_malware", "type",
+    "file", "filename", "file_name", "name", "hash", "md5", "sha1",
+    "sha256", "source", "category",
+}
+
+
+def normalize_name(value: object) -> str:
+    """Convert a source column name to a stable snake_case identifier."""
+    text = re.sub(r"[^0-9A-Za-z]+", "_", str(value).strip()).strip("_").lower()
+    return text or "unnamed_feature"
+
+
+def normalize_columns(columns: list[object]) -> tuple[list[str], dict[str, list[str]]]:
+    """Normalize names and disambiguate collisions by their source-column order.
+
+    The UCI files intentionally contain case- or punctuation-distinct fields such as
+    ``WORD``/``word`` and ``Import``/``__IMPORT_``. A simple lowercase snake-case
+    conversion collapses each pair. The three source files use the same relative
+    order for these fields, so an occurrence suffix produces stable shared names:
+    ``word``, ``word__2``, and so on.
+    """
+    counts: dict[str, int] = {}
+    normalized: list[str] = []
+    originals_by_base: dict[str, list[str]] = {}
+
+    for raw_column in columns:
+        base = normalize_name(raw_column)
+        counts[base] = counts.get(base, 0) + 1
+        occurrence = counts[base]
+        unique_name = base if occurrence == 1 else f"{base}__{occurrence}"
+        normalized.append(unique_name)
+        originals_by_base.setdefault(base, []).append(str(raw_column))
+
+    collisions = {
+        base: originals
+        for base, originals in originals_by_base.items()
+        if len(originals) > 1
+    }
+    if len(normalized) != len(set(normalized)):
+        raise ValueError("Column names are still duplicated after disambiguation.")
+    return normalized, collisions
+
+
+def find_required_file(filename: str) -> Path:
+    """Find one required CSV below data/raw, allowing an extracted subfolder."""
+    matches = list(RAW_DIR.rglob(filename)) if RAW_DIR.exists() else []
+    if len(matches) != 1:
+        raise FileNotFoundError(
+            f"Expected exactly one {filename!r} below {RAW_DIR}. "
+            "Follow data/README.md and check the extracted folder structure."
+        )
+    return matches[0]
+
+
+def load_source(source: str, filename: str, label: int) -> pd.DataFrame:
+    """Load one source file and attach provenance columns."""
+    frame = pd.read_csv(find_required_file(filename), low_memory=False)
+    normalized, collisions = normalize_columns(frame.columns.tolist())
+    frame.columns = normalized
+    # Consolidate the wide table before adding provenance columns. This avoids
+    # pandas fragmentation warnings on the 1,000+ column UCI source files.
+    frame = frame.copy()
+    frame.attrs["normalized_column_collisions"] = collisions
+    frame["source"] = source
+    frame["label"] = label
+    return frame
+
+
+def validate_collision_alignment(frames: dict[str, pd.DataFrame]) -> None:
+    """Ensure occurrence suffixes refer to the same raw fields in every source."""
+    collision_names = set().union(*(
+        set(frame.attrs.get("normalized_column_collisions", {}))
+        for frame in frames.values()
+    ))
+    for base in sorted(collision_names):
+        variants = {
+            source: tuple(frame.attrs.get("normalized_column_collisions", {}).get(base, []))
+            for source, frame in frames.items()
+        }
+        if len(set(variants.values())) != 1:
+            raise ValueError(
+                f"Normalized collision {base!r} is not aligned across source files: {variants}"
+            )
+
+    if collision_names:
+        print("Disambiguated normalized column collisions:")
+        reference = next(iter(frames.values())).attrs["normalized_column_collisions"]
+        for base in sorted(collision_names):
+            generated = [base] + [f"{base}__{index}" for index in range(2, len(reference[base]) + 1)]
+            print(f"  {reference[base]} -> {generated}")
+
+
+def main() -> None:
+    frames = {
+        source: load_source(source, filename, label)
+        for source, (filename, label) in FILES.items()
+    }
+    validate_collision_alignment(frames)
+
+    common = set.intersection(*(set(frame.columns) for frame in frames.values()))
+    candidate_features = sorted(common - FORBIDDEN_NAMES - {"source", "label"})
+    if not candidate_features:
+        raise ValueError("No shared candidate feature columns were found.")
+
+    combined_parts = []
+    for source, frame in frames.items():
+        part = frame[candidate_features].copy()
+        part.insert(0, "source", source)
+        part.insert(1, "label", int(FILES[source][1]))
+        combined_parts.append(part)
+    combined = pd.concat(combined_parts, ignore_index=True)
+
+    numeric = combined[candidate_features].apply(pd.to_numeric, errors="coerce")
+    numeric = numeric.replace([np.inf, -np.inf], np.nan)
+    usable_ratio = numeric.notna().mean()
+    retained = usable_ratio[usable_ratio >= 0.95].index.tolist()
+    numeric = numeric[retained]
+
+    non_constant = numeric.nunique(dropna=True)
+    retained = non_constant[non_constant > 1].index.tolist()
+    numeric = numeric[retained]
+    if numeric.empty:
+        raise ValueError("No usable numeric features remained after validation.")
+
+    prepared = pd.concat([combined[["source", "label"]], numeric], axis=1)
+    prepared = prepared.sample(frac=1.0, random_state=42).reset_index(drop=True)
+    prepared.to_csv(OUTPUT_PATH, index=False)
+
+    manifest = pd.DataFrame({
+        "feature": retained,
+        "non_missing_ratio": numeric[retained].notna().mean().values,
+        "unique_values": numeric[retained].nunique(dropna=True).values,
+        "feature_family": "unclassified",
+    })
+    manifest.to_csv(MANIFEST_PATH, index=False)
+
+    print(f"Created {OUTPUT_PATH}")
+    print(f"Shape: {prepared.shape}")
+    print("Class counts:")
+    print(prepared["label"].value_counts().sort_index())
+    print("Source counts:")
+    print(prepared["source"].value_counts())
+    print(f"Created {MANIFEST_PATH}")
+
+
+if __name__ == "__main__":
+    main()