Imbalanced Data

A fraud team at a payments company trains a random forest on 100,000 labelled transactions. Training accuracy comes back at 99.8%, validation accuracy at 99.7%. They congratulate each other, ship the model, and move on. Three weeks into production they notice something strange: the model has flagged zero fraudulent transactions. On inspection it is correctly predicting “not fraud” for every single transaction it sees — and getting 99.7% accuracy by doing so, because 99.7% of transactions really are not fraud. The real fraud, 0.3% of traffic, has cost the company £2.3M in three weeks, and the model has done nothing to catch it.

The model was not buggy. It optimised exactly the quantity it was told to optimise — accuracy — and it chose the rational strategy for that objective on a heavily imbalanced dataset: always predict the majority class. This chapter is about recognising that accuracy is the wrong metric for imbalanced classification and knowing what to measure instead.

The accuracy paradox

With any imbalance above roughly 80/20, accuracy becomes uninformative. On an 8% default rate loan dataset, predicting “never defaults” for every applicant gets 92% accuracy. On a 1% fraud dataset, predicting “not fraud” gets 99%. These numbers look like model quality but they measure only how imbalanced the data was.

The lesson is not that accuracy is useless — it is that accuracy conflates two entirely different questions: ‘did the model get the majority class right?’ (almost always yes if there is an imbalance) and ‘did the model get the minority class right?’ (usually what you actually care about). For imbalanced problems the right metrics separate these two questions.

The confusion matrix is the primary diagnostic

Every classification metric starts from the confusion matrix — a 2×2 table counting how many examples fell into each combination of (actual, predicted) for a binary problem.

From the four counts you derive every relevant metric:

• Accuracy = (TP + TN) / total. For imbalanced data this is dominated by the large TN count, so it says little.
• Precision = TP / (TP + FP). ‘Of the alerts I raised, how many were real?’ At 11 TP and 20 FP: 35%. Only a third of flagged applicants were actual defaulters.
• Recall = TP / (TP + FN). ‘Of the real defaulters, how many did I catch?’ At 11 TP and 5 FN: 69%. The model catches about two-thirds of actual defaulters.
• F1 = 2 · Precision · Recall / (Precision + Recall). The harmonic mean of precision and recall. At 0.35 and 0.69: F1 = 0.46. Useful as a single summary but it hides the trade-off between the two.

The precision/recall pair is the core metric pair for imbalanced classification. They often conflict: a more aggressive classifier (more things flagged as positive) catches more real positives but includes more false alarms, so recall goes up and precision goes down.

PR curves vs ROC curves

As you sweep the classifier’s decision threshold from 0 to 1, both precision and recall change — and so does another pair, true positive rate (= recall) and false positive rate (FP / (FP + TN)). These produce two different curves when plotted:

The mechanism: in imbalanced data the negative class is much bigger than the positive class, so the FPR denominator (TN + FP) stays large even when FP is substantial. Small changes in FP move FPR very little, which inflates every point on the ROC curve. PR curves use precision, whose denominator (TP + FP) depends only on predicted positives — a much tighter quantity. On a problem where imbalance exceeds roughly 10:1, prefer PR-AUC as your single headline metric over ROC-AUC.

Three knobs for rebalancing

Once you have the right metrics, the question becomes ‘how do I train a model that scores well on them?’ There are three operational approaches.

Threshold adjustment. Classifiers typically output a probability; the usual 0.5 decision threshold is arbitrary and rarely optimal for imbalanced data. Lowering the threshold (say, to 0.3) catches more positives at the cost of more false positives — higher recall, lower precision. Raising it does the opposite. This is the cheapest intervention and often sufficient: just pick the threshold that maximises the metric you care about on the validation set. The playground in this chapter lets you drag the threshold and watch the confusion matrix recompute live.

Class weighting in the loss. Most training algorithms accept a per-class weight. Setting the minority-class weight to, say, 12× the majority weight tells the optimiser to treat each minority example as if it appeared 12 times. This shifts the model’s learned decision boundary toward catching more of the minority class. In scikit-learn this is the class_weight='balanced' parameter (auto-scales by inverse frequency); in neural networks it appears in the cross-entropy loss as explicit weight factors.

Resampling — oversample minority, undersample majority, or SMOTE. Rather than re-weighting the loss, physically change the training-data distribution before fitting. Random oversampling duplicates minority examples; random undersampling discards majority examples; SMOTE (Synthetic Minority Over-sampling TEchnique) generates new minority examples by interpolating between real ones and their k-nearest minority neighbours. Resampling is popular because it works with any algorithm that does not support class weights natively, but it has risks — duplicated examples encourage overfitting, SMOTE can generate synthetic examples that do not resemble real data.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, precision_recall_fscore_support
from imblearn.over_sampling import SMOTE  # pip install imbalanced-learn

# X: 10,000 × 50 loan features; y: binary default label (8% positive rate)
X_train, X_test, y_train, y_test = train_test_split(
  X, y, test_size=0.2, random_state=42, stratify=y,
)

# Option A: class weighting — no data changes
model_a = LogisticRegression(class_weight='balanced', max_iter=1000)
model_a.fit(X_train, y_train)

# Option B: SMOTE oversampling — inflate minority to match majority
smote = SMOTE(random_state=42)
X_train_bal, y_train_bal = smote.fit_resample(X_train, y_train)
model_b = LogisticRegression(max_iter=1000)
model_b.fit(X_train_bal, y_train_bal)

# Option C: threshold adjustment on a plain model
model_c = LogisticRegression(max_iter=1000)
model_c.fit(X_train, y_train)

def evaluate(model, label, threshold=0.5):
  probs = model.predict_proba(X_test)[:, 1]
  preds = (probs >= threshold).astype(int)
  tn, fp, fn, tp = confusion_matrix(y_test, preds).ravel()
  prec, rec, f1, _ = precision_recall_fscore_support(
      y_test, preds, average='binary', zero_division=0,
  )
  print(f"{label:22s} prec={prec:.2f}  rec={rec:.2f}  F1={f1:.2f}  (tp={tp}, fp={fp}, fn={fn})")

evaluate(model_a, 'class-weighted',       threshold=0.5)
evaluate(model_b, 'SMOTE-oversampled',    threshold=0.5)
evaluate(model_c, 'plain, thr=0.3',       threshold=0.3)
evaluate(model_c, 'plain, thr=0.5 default', threshold=0.5)

# Typical output — all three rebalancing approaches dominate the default:
# class-weighted         prec=0.31  rec=0.68  F1=0.43  (tp=109, fp=245, fn=51)
# SMOTE-oversampled      prec=0.33  rec=0.70  F1=0.45  (tp=112, fp=230, fn=48)
# plain, thr=0.3         prec=0.30  rec=0.72  F1=0.42  (tp=115, fp=268, fn=45)
# plain, thr=0.5 default prec=0.72  rec=0.14  F1=0.23  (tp=22,  fp=8,   fn=138)

The final row in that output is the uncorrected baseline — 72% precision but 14% recall means the model is nearly useless for catching real defaulters. The three remedies above all push recall above 65% while keeping precision reasonable. In practice you would run this comparison on your own data and pick the approach that scores best on your chosen metric (F1 if you are neutral, recall if false negatives are expensive, precision if false alarms are).

Where this chapter sits

This is the last chapter of the Data Splits and Evaluation topic. Everything so far — train/val/test splits, cross-validation, overfitting diagnostics — assumed that accuracy on the validation set was the quantity to optimise. For imbalanced data that assumption breaks, so this chapter replaces ‘accuracy’ with a small family of metrics (precision, recall, F1, PR-AUC) appropriate for rare-positive prediction. With those in place, everything from the earlier chapters still applies: you still split before training, you still cross-validate if the data is small, you still diagnose overfitting by the gap between training and validation performance. The only change is that ‘performance’ is now a precision/recall pair rather than a single accuracy number.

Domain 2 closes with this chapter. The natural next step is Classical ML — the first domain where learners actually see algorithms in action: linear regression, logistic regression, decision trees, random forests, k-NN. Each one builds on the data-preparation foundation laid across Domain 2, and each has its own interaction with the imbalance and overfitting issues discussed in this topic.