Train, Validation, and Test Splits

The risk team at a regional bank has 10,000 labelled loan applications. Each application has a few dozen features — income, employment length, debt-to-income ratio, postcode, credit history — and a binary label: did the applicant default within twelve months. The team is evaluating three models: logistic regression, a decision tree, and a gradient-boosted classifier. Every model reports accuracy above 95% on the full 10,000-row dataset. The senior analyst asks which model to deploy.

The correct answer is that none of those numbers measure anything useful. The models were trained on the same data they are now being evaluated on. A decision tree deep enough can drive its training accuracy to exactly 100% on any dataset by treating the tree as a lookup table from inputs to memorised labels. That does not mean it will work on next month’s applicants. It means it memorised this month’s.

Why Training Accuracy Means Nothing

Training a classifier is an optimisation: the model’s parameters are adjusted until the error on the training data is as low as possible. Every weight, every split threshold, every leaf value exists because it reduced the error on examples the model has now seen many times. Reporting that error as “accuracy” is circular — the model is at a minimum of its own objective on those specific rows, and that minimum depends on the data more than on the underlying pattern.

This circularity is not a subtle theoretical issue. A decision tree with no depth limit on the bank’s 10,000 applications can simply learn a split for every single row: a tree with 10,000 leaves, one per applicant, each labelled with that applicant’s true outcome. Training accuracy: 100%. Accuracy on the next 1,000 new applicants: roughly the base rate — no better than a coin weighted by the class proportion. The tree did not learn to predict defaults. It memorised the labels.

The only honest way to measure whether a model generalises is to evaluate it on data it has never seen during training. That data has to be set aside — held out — before any fitting begins. Split first, then train. Never the reverse.

Three Splits, Three Purposes

A single held-out set is enough to measure final performance, but running a real ML project involves many decisions that each need their own held-out measurement: which features to include, which model family to try, what hyperparameters to set, how much regularisation to use. If every one of those decisions is made by looking at the same held-out set, the set loses its independence — after fifty hyperparameter tweaks evaluated on the same 1,500 rows, the model has been implicitly fitted to those 1,500 rows even though it was never trained on them. The held-out score becomes optimistic.

The standard solution is to split the data into three parts:

• Training set — the data the model actually learns from. Weights get adjusted, tree splits get chosen, parameters get fitted. The larger this set, the more patterns the model can potentially capture.
• Validation set — a workbench for comparing models and tuning hyperparameters. Look at it as often as you need while making decisions. Different feature sets, different learning rates, different regularisation strengths are all evaluated here.
• Test set — a held-out envelope, sealed until the end. Opened exactly once, after every decision about the model is finalised. The test accuracy is the honest answer to how well the model generalises.

The rule that matters most: every choice that changes the model based on a held-out measurement must happen before the test score is seen. Compare a dozen hyperparameter settings on the validation set, pick the best, retrain on training + validation, evaluate once on the test set, report that number. Going back to try something else after seeing the test score breaks the honesty of the measurement.

A useful mental model: the validation set is the pencil draft, where you work out the solution. The test set is the final-exam answer sheet — visible only after you have committed to what you believe is the right answer. Peeking at the answer sheet and then changing your working is a different activity altogether.

Data Leakage: The Silent Failure

The subtlest way to break an honest evaluation is not by explicitly training on test data — most practitioners catch that — but by letting information from the test set influence the training process indirectly. This is called data leakage, and it turns an honest 85% test accuracy into an optimistic 92% that dissolves the moment the model sees production data.

The classic case: feature scaling. A StandardScaler computes a mean and standard deviation and uses them to centre and normalise every row. If the scaler is fitted on the full 10,000-row dataset before splitting, those statistics were computed partly from the 1,500 rows now sitting in the test set. The model trains on features whose standardisation the test set helped determine. The test set never appears in the training loss, but its distribution has leaked in through the scaler.

Scaler leakage is the most visible form, but many variants exist:

• Feature engineering using the full dataset. Computing a feature like “customer’s percentile within the dataset’s income distribution” uses every applicant’s income to compute the percentile — including the test applicants. The feature values for training rows now encode information about test rows.
• Target encoding without cross-validation. Replacing a categorical variable with the mean target value per category computes those means from all rows, including the held-out ones. The training model gets to see an aggregate signal from test labels.
• Imputation using the full dataset mean. Filling missing values with the overall dataset mean lets test-row features shift the mean, which then fills missing training-row values with a statistic that already reflects the test set.
• Temporal leakage. For time-ordered data — sales by day, sensor readings, user actions — a random split mixes past and future rows. The model trains on some future rows and is then evaluated on past ones. Production-time performance, which can only use past data, will be worse than the test score suggests.
• Grouped data leakage. If the dataset has multiple rows per patient or multiple frames per video, a random split can put different rows from the same patient in both training and test. The model has now “seen” that patient during training and will predict their test rows more accurately than it could a genuinely new patient.

The defensive habit is to think of every preprocessing step as a model of its own. A scaler is a model: it has parameters (mean and standard deviation) that are fitted to data. An imputer is a model. A feature-encoder is a model. Every fitted-then-applied transformation must be fitted on the training set only and then applied — without refitting — to validation and test. In scikit-learn this is enforced by wrapping the scaler and the classifier together in a Pipeline:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# X, y: 10,000 labelled loan applications
# X shape (10_000, n_features), y shape (10_000,) with 0 = repaid, 1 = defaulted

# Step 1: split BEFORE any preprocessing
X_train, X_temp, y_train, y_temp = train_test_split(
  X, y,
  test_size=0.30,
  random_state=42,
  stratify=y,           # preserves the 8% default rate in both halves
)
X_val, X_test, y_val, y_test = train_test_split(
  X_temp, y_temp,
  test_size=0.50,
  random_state=42,
  stratify=y_temp,
)

# Step 2: bundle scaler and model so the scaler learns only from training data
pipe = Pipeline([
  ('scaler', StandardScaler()),
  ('model',  LogisticRegression(C=1.0, max_iter=1000)),
])

# Step 3: fit on training only — the scaler's mean and std are training statistics
pipe.fit(X_train, y_train)

# Step 4: evaluate on validation during development
val_acc = pipe.score(X_val, y_val)

# Step 5: after every decision is final, evaluate once on test
test_acc = pipe.score(X_test, y_test)
print(f"Val accuracy:  {val_acc:.3f}")
print(f"Test accuracy: {test_acc:.3f}")

The pipeline does more than save typing. It makes leakage impossible at the scaler step: pipe.fit(X_train, y_train) cannot accidentally see validation or test rows, because the call was made with only X_train.

Choosing Split Ratios

Once the three-split principle is in place, the remaining question is how much data to put in each split. The right answer depends on the dataset’s absolute size, not on a fixed percentage.

A practical constraint for classification: every split must contain enough examples of the rare class for the metric to be meaningful. At an 8% default rate, a validation set of 100 rows has roughly 8 defaulters — and a single false negative shifts the false-negative rate by 12 percentage points. When the rare class is what you actually care about predicting, plan the split around that class’s absolute count, not the total size.

Two observations from the learning curve tell you something useful about your project. First, if training and validation error converge at your actual dataset size — as they do here at n ≈ 5,000 — adding more data will not significantly improve the model. Effort is better spent on features or model architecture. Second, if there is still a large gap at your dataset size, more data is likely to help more than more model capacity; the model is overfitting because its training set is too small for what it needs to learn.

Why This Chapter Is the Foundation

Everything in the rest of this topic rests on the three-split rule and the leakage-free pipeline. Cross-validation, covered next, is an extension of the validation split designed for smaller datasets — it repeats the split many times rather than relying on a single one. Overfitting and underfitting, the chapter after that, are defined relative to the gap between training and held-out error — the learning curve above is a first look at that gap. Class imbalance — the chapter after — starts with the problem that split ratios alone are not enough when the rare class is what you actually need to measure.

If you take only one habit away: split the data first, fit every preprocessing step on the training portion only, and open the test envelope exactly once. The rest is mechanics.