Overfitting and Underfitting

A data scientist trains a decision tree on 5,000 bank loan applications. Training accuracy comes back at 99.8%. They are delighted — until the validation accuracy prints 71%. Something is wrong. The same scientist, a week later, trains a logistic regression on the same data. Training accuracy 74%, validation accuracy 73%. Both models are broken but in opposite ways. The tree memorised the training set — huge train accuracy, poor generalisation. The logistic regression is too simple to capture the pattern — both accuracies are low. This chapter is about recognising which failure you are looking at and what to do about each one.

Three regimes

Every trained model sits in one of three regimes, diagnosed by the gap between training and validation error.

These three regimes correspond to three diagnostic patterns:

• Underfit: training error high, validation error high (and roughly equal). The model is too simple to represent the data.
• Good fit: training error low, validation error low (and roughly equal). Training and validation error tell the same story.
• Overfit: training error low, validation error high. The model has fitted the noise, not the pattern.

The gap val_err − train_err is the direct diagnostic. Small or zero is good. Large is overfitting. If both numbers are large, the model is underfit and the gap may still be small — the failure is absolute, not relative.

The complexity curve — what model size actually buys you

As you increase model capacity (polynomial degree, tree depth, network width), training error falls monotonically — more parameters can always fit more of the training data, right down to zero error if capacity is unbounded. Validation error does not behave this way. It follows a characteristic U-shape: it falls as the model captures real structure, then rises as the model picks up noise-level features that do not generalise.

The shape of the validation curve is the bias-variance trade-off made visible.

• Bias (high at low degree): the model’s structural assumption is too restrictive. It cannot represent the true pattern regardless of how much data you give it.
• Variance (high at high degree): the model’s predictions change dramatically when you resample the training data, because it has enough flexibility to chase whatever noise is present.

Bias falls as capacity rises; variance rises as capacity rises. Total validation error is roughly their sum, so the minimum sits somewhere in the middle. The practical goal of modelling is not to minimise bias (overfit) or minimise variance (underfit) but to find the capacity that minimises their sum.

Three knobs for fixing overfit models

When you diagnose overfitting — training error low, validation error much higher — you have three main interventions available.

More training data. Adding data makes memorisation progressively harder. A model with fixed capacity facing a larger dataset must abstract — there are too many examples to fit individually. This is often the most effective remedy, and often the most expensive. When you cannot get more real data, sometimes augmentation (slight transformations of existing examples) or semi-supervised learning from unlabelled data can approximate it.

Less capacity. Directly reduce the model’s expressive power — fewer tree splits, shallower network, lower polynomial degree. This is cheap but requires you to know which capacity is too much. A common pattern: start with a model you suspect is too small and increase capacity until training and validation error diverge, then back off slightly.

Regularisation. Add a penalty term to the loss that encourages simpler solutions without changing the nominal model size. The most common form is L2 (ridge) regularisation, which adds λ times the sum of squared coefficients to the loss:

Large coefficients now cost the optimiser more, so it picks the smallest coefficients that still fit the data well. Empirically this produces much smoother fits without removing any model parameters.

L1 regularisation (lasso) uses |cᵢ| instead of cᵢ². It tends to drive many coefficients to exactly zero, effectively removing features from the model — so it doubles as an automatic feature-selection method. For feature-sparse problems it is sometimes preferable to L2.

Three knobs for fixing underfit models

Underfitting is the opposite problem and needs the opposite interventions.

More capacity. The model’s structural assumption is too restrictive. Give it more parameters to work with: higher polynomial degree, deeper tree, wider network. This is the first thing to try when both train and validation error are high.

Better features. If the available features do not contain the signal, no capacity will help. Go back to feature engineering: add interactions, nonlinear transformations, external data sources. Sometimes underfit is actually an information problem disguised as a capacity problem.

Weaker regularisation. If you inherited a model with heavy regularisation (large λ) and it is underfit, lower λ and the model can use more of its nominal capacity. Sometimes this is done by accident — a default regularisation value appropriate for a small dataset is too strong when you move to a larger one.

Cross-validation closes the diagnostic loop

The previous chapter introduced cross-validation as a way to get a reliable validation error estimate on small datasets. This chapter is what CV scores feed into: once you have a reliable estimate of validation error, you use it to pick the model capacity that minimises it. That is the standard hyperparameter-tuning workflow.

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler

# X: 1-D training data reshaped to (n, 1); y: target values
# Sweep polynomial degree from 1 to 15 and report CV error

degrees = list(range(1, 16))
scores = []
for d in degrees:
  pipe = Pipeline([
      ('poly',  PolynomialFeatures(degree=d)),
      ('scale', StandardScaler()),
      ('model', Ridge(alpha=0.0)),     # no regularisation here
  ])
  cv = cross_val_score(pipe, X, y, cv=5, scoring='neg_mean_squared_error')
  scores.append(-cv.mean())

best_idx = int(np.argmin(scores))
print(f"Best degree: {degrees[best_idx]}, CV MSE: {scores[best_idx]:.4f}")
print(f"All scores:  {np.array(scores).round(3).tolist()}")

# Typical output on the chapter's data:
# Best degree: 3, CV MSE: 0.0512
# Scores decrease sharply from d=1 to d=3, then rise steadily

# Now sweep ridge alpha at the chosen degree — regularisation tuning
alphas = [0, 0.01, 0.1, 0.3, 1.0, 3.0, 10.0]
for alpha in alphas:
  pipe.set_params(model__alpha=alpha)
  cv = cross_val_score(pipe, X, y, cv=5, scoring='neg_mean_squared_error')
  print(f"  alpha={alpha:6.2f}  CV MSE={-cv.mean():.4f}")

The whole workflow — sweep capacity, pick the minimum, verify on test — is what the playground makes interactive. Drag the polynomial degree from 1 to 15 and you will watch the validation MSE trace out the U-shape directly; drag the regularisation slider and you will see a high-degree fit smooth out in real time. The next chapter takes us out of the regression world and into classification, where class imbalance creates its own family of overfit-adjacent failures.