labs/Lab-4/Lab-4-Practical/05_Probability_Thresholds_Interpretation_Lab.ipynb

Module 4 — Lab 5: Probability, Thresholds, and Interpretation

Topics: Logistic Regression, predict_proba, validation, thresholds, coefficients, actions

Lab version: Complete every TODO, run the cells in order, and use the self-check before opening the Solution.

Learning objectives

After completing this lab, you will be able to:

• reserve separate training, validation, and test partitions;
• convert Logistic Regression probabilities into decisions;
• choose a threshold using validation data only;
• measure the threshold trade-off between false positives and false negatives;
• interpret influential terms through model coefficients;
• map model scores to allow, analyst-review, and quarantine actions.

In [ ]:

from pathlib import Path
import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

DATA_PATH = Path("data/SMSSpamCollection")
RANDOM_STATE = 42
THRESHOLDS = np.arange(0.20, 0.81, 0.05)

if not DATA_PATH.exists():
    raise FileNotFoundError(
        "Missing data/SMSSpamCollection. Follow data/README.md to download the official dataset."
    )

1. Prepare train, validation, and test data

Training fits model parameters, validation selects the threshold, and test estimates final performance. The test labels must not influence threshold choice.

In [ ]:

df = pd.read_csv(DATA_PATH, sep="\t", header=None, names=["label", "message"])
df["label"] = df["label"].astype("string").str.strip().str.lower()
df["message"] = df["message"].astype("string").str.strip()
df = df.dropna(subset=["label", "message"])
df = df[df["label"].isin(["ham", "spam"]) & df["message"].ne("")]
conflict_counts = df.groupby("message")["label"].nunique()
df = df[~df["message"].isin(conflict_counts[conflict_counts.gt(1)].index)]
df = df.drop_duplicates(subset=["label", "message"]).reset_index(drop=True)

X = df["message"]
y = df["label"].map({"ham": 0, "spam": 1}).astype(int)

In [ ]:

# TODO 1: split the data into 60% training and 40% temporary data.
X_train, X_temp, y_train, y_temp = None, None, None, None

# TODO 2: split the temporary data equally into validation and test sets.
X_valid, X_test, y_valid, y_test = None, None, None, None

print("Train / validation / test:", None)
print("Spam rates:", None)

2. Train a probabilistic text classifier

Logistic Regression produces a spam probability for each message. A threshold converts that continuous score into a binary prediction.

In [ ]:

# TODO 3: create and fit a TF-IDF + Logistic Regression pipeline.
lr_pipeline = None

# TODO 4: calculate validation spam probabilities.
valid_probabilities = None

print("First probabilities:", None)

3. Select a threshold on validation data

The classroom policy requires at least 80% spam recall, then prefers the qualifying threshold with the highest spam precision. This is an explicit decision rule, not a universal production policy.

In [ ]:

# TODO 5: evaluate every threshold in THRESHOLDS on validation data.
threshold_table = None

# TODO 6: among thresholds with spam recall >= 0.80, select the one
# with the highest precision. If none qualify, use the highest F1.
selected_threshold = None

print(threshold_table.round(4))
print("Selected threshold:", selected_threshold)

4. Evaluate once on the untouched test set

In [ ]:

# TODO 7: evaluate the selected threshold once on the untouched test set.
test_probabilities = None
test_predictions = None
test_cm = None
test_metrics = None

print(pd.Series(test_metrics).round(4))
print(test_cm)

5. Interpret influential terms

Positive coefficients push the decision toward spam; negative coefficients push it toward ham. They describe model behavior and association, not causal truth.

In [ ]:

# TODO 8: extract the fitted vocabulary and Logistic Regression coefficients.
feature_names = None
coefficients = None

# TODO 9: create a table with the 15 strongest ham terms and
# the 15 strongest spam terms.
top_features = None
top_features

6. Map probabilities to actions

Use fixed demonstration bands: below 0.30 is allow, 0.30–0.70 is analyst_review, and 0.70 or above is quarantine. A production policy would set these boundaries from business cost, capacity, and validation evidence.

In [ ]:

# TODO 10: map test probabilities to allow, analyst_review, or quarantine.
policy_actions = None
action_summary = None

action_summary

Self-check

In [ ]:

assert len(X_train) + len(X_valid) + len(X_test) == len(df)
assert set(lr_pipeline.named_steps) == {"tfidf", "model"}
assert np.all((valid_probabilities >= 0) & (valid_probabilities <= 1))
assert threshold_table.index.is_monotonic_increasing
assert selected_threshold in threshold_table.index
assert len(test_predictions) == len(y_test)
assert test_cm.shape == (2, 2)
assert len(top_features) == 30
assert set(top_features["direction"]) == {"ham", "spam"}
assert set(policy_actions.astype("string").unique()).issubset({"allow", "analyst_review", "quarantine"})
print("✓ Lab 5 completed")

Reflection questions

1. Why is a validation set needed when selecting a threshold?
2. What usually happens to spam recall when the threshold is lowered?
3. Why are large positive coefficients not proof that a term causes spam?