Cross-Validation

A regional bank branch has 800 labelled loan applications — smaller than the 10,000-applicant dataset from the previous chapter. The analyst tries a 70/15/15 split: 560 for training, 120 for validation, 120 for test. At an 8% default rate, the 120-applicant validation set contains roughly 10 defaulters. The false-negative rate — the metric the credit risk committee cares about — is being estimated from 10 examples.

The analyst runs the split twice with different random seeds. The first gives 91% accuracy. The second gives 86%. Same model. Same 800 applicants. A 5-percentage-point swing from a different random seed.

Which number goes to the committee?

The Problem With One Split

With 10,000 applicants, a 15% validation set gives 1,500 applicants — enough for reliable estimates. With 800 applicants, 15% is 120 — and at an 8% default rate, that yields roughly 10 defaulters. One misclassification shifts the false-negative rate by 10 percentage points.

The split problem is not just about defaulters. The entire 120-applicant validation set might, by chance, be drawn from a period when the bank’s lending standards were stricter — making those applicants systematically easier or harder to classify than the training applicants. A different random seed draws a different period. The measured accuracy reflects the luck of the draw, not the quality of the model.

Run the same 800 applicants through the same logistic regression with seed 42: you get 91% validation accuracy. Swap to seed 137: you get 86%. The model has not changed. The training data has not changed. Only which 120 applicants happened to end up in the held-out set changed. That 5-point swing is sampling noise masquerading as model performance. Reporting either number in isolation is misleading.

How k-Fold Works

Instead of one fixed split, divide the 800 applications into k equal folds. Train the model on k−1 folds, validate on the remaining fold, then repeat k times — each time holding out a different fold. Every application is used for validation exactly once. Average the k accuracy scores.

A worked example: 5-fold cross-validation on the 800-application dataset.

The ± 0.9% standard deviation across folds measures estimate reliability. A single split gave either 91% or 86% depending on which 120 applicants ended up held out. Five-fold CV gives 88.1% ± 0.9% — a tighter, more trustworthy number derived from five independent evaluations of the same model on different held-out subsets.

The test set is still held out and untouched. Cross-validation replaces the validation split — not the test split. The correct pipeline:

1. Split off the test set first — 120 applicants, never touched until final evaluation.
2. Use CV on the remaining 680 applicants to tune and evaluate the model.
3. Evaluate the final model on the test set once, after all CV-based decisions are final.

The CV score estimates how well the model generalises. The test score measures it honestly, once.

Choosing k and Stratification

The choice of k involves a trade-off between estimate reliability and computational cost.

For the bank dataset at 800 applications, k=5 or k=10 are both reasonable. k=5 is faster — five model training runs instead of ten. k=10 gives slightly lower estimate variance because ten fold scores are averaged instead of five. The practical rule: k=5 for datasets under 5,000 examples, k=10 for larger datasets.

Stratified k-fold matters here because 8% of applicants default — 64 defaulters out of 800. A regular 5-fold split might assign 10 defaulters to one fold and 16 to another by chance. With 10 defaulters in a validation fold, the false-negative rate estimate from that fold is almost meaningless. Stratified k-fold preserves the 8% default rate in every fold, guaranteeing roughly 13 defaulters per fold.

For any imbalanced classification problem, stratified k-fold is the default — not an optional refinement. Using regular k-fold with an 8% minority class introduces fold-to-fold variance in minority class counts that has nothing to do with model quality. Stratification eliminates that source of noise.

Leave-One-Out and When to Use It

Leave-one-out (LOO) is k-fold where k equals n. Each of the 800 applications is held out once. The model trains on the remaining 799 each time, for a total of 800 training runs.

The advantage: each training set contains 799 of 800 available examples — the model sees nearly all the data in every fold. The estimate variance approaches zero because 800 fold scores are averaged. The disadvantage is cost: 800 training runs for a dataset of 800 examples. For most models this is computationally prohibitive.

LOO is appropriate when:

• The dataset has fewer than 50–100 examples, making any ordinary fold too small to evaluate performance.
• The model trains very fast — linear regression, small decision trees — so hundreds of training runs are affordable.
• The context demands minimum bias in the estimate — medical trials, rare event datasets, scientific reproducibility.

For the bank’s 800-application dataset with logistic regression, LOO requires 800 training runs. k=10 requires ten. The estimate quality improvement from k=10 to LOO is typically less than 0.5 percentage points for most models. The compute cost multiplies by 80. Use LOO when training is cheap and the dataset is very small.

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# All 800 applications: X (features), y (default label, binary)

# Step 1: Separate the test set before any CV — never touched until the end
X_trainval, X_test, y_trainval, y_test = train_test_split(
  X, y,
  test_size=0.15,
  random_state=42,
  stratify=y           # preserves 8% default rate in both halves
)

# Step 2: Build a pipeline so the scaler is fitted inside each fold
# This prevents preprocessing leakage across folds
pipe = Pipeline([
  ('scaler', StandardScaler()),
  ('model', LogisticRegression(C=1.0, max_iter=1000))
])

# Step 3: Stratified 5-fold CV on the 680 non-test applicants
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

cv_scores = cross_val_score(
  pipe, X_trainval, y_trainval,
  cv=skf,
  scoring='accuracy'
)

print(f"CV scores:  {cv_scores.round(3)}")
print(f"Mean ± std: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# cross_val_score does NOT touch X_test or y_test — the test set is safe

# Step 4: Retrain on all non-test data, then evaluate once on the test set
pipe.fit(X_trainval, y_trainval)
test_accuracy = pipe.score(X_test, y_test)
print(f"Test accuracy: {test_accuracy:.3f}")

The pipeline wraps the scaler and the model together. cross_val_score refits the entire pipeline in each fold — scaler included — so the fold’s validation applicants are never incorporated into the scaler’s stored statistics. This prevents a subtle form of preprocessing leakage inside the CV loop itself.