Other Reduction Techniques

A music platform has a catalogue of 10,000 songs, each described by 128 audio features produced by a neural network. They want a 2D visualisation to check whether the features naturally separate songs by genre. The first attempt uses PCA: the top two principal components are plotted as x and y axes, songs coloured by genre. The result is disappointing — a single elongated cloud with all four genres overlapping. It looks like the features are not good enough.

They switch to UMAP with default parameters. Same songs, same features. Four clearly separated clusters, one per genre, with jazz and classical adjacent and pop and rock adjacent. The features were fine all along. PCA’s linear assumption was the problem: the genre separation lives along a curved manifold in the 128-dimensional space, and no straight line through the origin captures it. UMAP’s non-linear method finds it easily.

This chapter is about when PCA is not enough and what to use instead.

Why PCA can miss structure

PCA is fundamentally a rotation. It finds the best linear subspace of the feature space — the k-dimensional plane that captures the most variance. If the interesting structure in the data is also linear, PCA is essentially optimal. If it is curved, PCA’s projection will average over the curvature and the top components may not reveal it.

A classic failure is the Swiss roll: 3D points arranged on a curled-up 2D sheet. The intrinsic structure is 2D (the uncurled sheet), but the sheet curves through 3D space. PCA on the Swiss roll produces a top-2D projection that flattens the curl by collapsing the two ends of the sheet on top of each other — the unrolled structure is destroyed. A non-linear manifold-learning method can recover the sheet.

Audio features, image embeddings, and word embeddings from neural networks tend to have curved structure for the same reason: the features encode similarity via learned nonlinear transformations. Linear projection often does not respect those similarities.

t-SNE: preserving local neighbourhoods

t-SNE (t-distributed Stochastic Neighbour Embedding) is specifically designed for visualisation. Its optimisation objective: find a 2D layout in which every point’s probability distribution over its nearest neighbours matches the corresponding distribution in the original high-dimensional space. It is explicitly preserving local structure — who is near whom — and not trying to preserve global distances.

The core parameter is perplexity: how many neighbours each point attempts to preserve. It acts like a smoothed version of k in k-nearest-neighbours. At perplexity 5, each point considers about 5 neighbours, which produces tight small clusters. At perplexity 80, each point considers many neighbours, which produces larger merged regions. The practical range is 5 to 50, with a default of 30.

Three properties of t-SNE that you need to know:

• Distances between clusters are not meaningful. t-SNE’s loss does not preserve inter-cluster distances, only local neighbourhoods. Two clusters that are far apart in the original high-dimensional space might end up close in the t-SNE layout, or vice versa. Use t-SNE to answer ‘are there clusters?’, not ‘how are the clusters arranged relative to each other?’
• Results depend on random initialisation. Different random seeds produce different layouts that all have similar clusters but different orientations. Do not build anything downstream that depends on the specific coordinate values.
• t-SNE is slow. On datasets above 10,000 points the runtime becomes inconvenient (tens of minutes to hours). UMAP is usually faster and gives comparable or better visualisations.

UMAP: faster and more global

UMAP (Uniform Manifold Approximation and Projection) was introduced as a near-drop-in replacement for t-SNE that addresses several of its limitations. It is based on a topological theory (simplicial complexes on a fuzzy graph — the mathematical detail is not required to use it) but the practical upshot is:

• Faster. Runtime is roughly O(n log n) instead of O(n²). On 100,000 points, UMAP takes minutes; t-SNE takes hours.
• Preserves more global structure. Inter-cluster distances in a UMAP plot are more trustworthy than in t-SNE — clusters that are genuinely close in the original space tend to be close in the UMAP layout.
• Principled defaults. The main parameters are n_neighbors (the local neighbourhood size, roughly equivalent to perplexity — default 15) and min_dist (how tightly points within a cluster pack together — default 0.1). The defaults work well for a wide range of datasets with minimal tuning.

For most current applications where you would have used t-SNE a few years ago, UMAP is now the default choice. The practical workflow is: try UMAP with default parameters. If the visualisation is not informative, increase n_neighbors to 30 or 50 to emphasise global structure, or decrease it to 5 to see more local detail.

LDA: supervised class separation

All three methods above (PCA, t-SNE, UMAP) are unsupervised — they ignore any labels the data might have. Linear Discriminant Analysis (LDA) does the opposite: it uses labels explicitly to find the directions that best separate classes.

LDA finds directions that maximise the ratio of between-class variance to within-class variance. Along these directions, the class means are as far apart as possible while the within-class spread is as small as possible — exactly what a downstream linear classifier wants. For a C-class problem, LDA produces at most C − 1 components (for 3 classes, at most 2 LDA axes).

LDA’s constraints are:

• Requires labels. Unlike PCA, you cannot apply it without a target variable.
• Linear. It is a linear method and will fail on non-linearly separable data just like PCA.
• Gaussian class assumption. LDA assumes each class has a Gaussian distribution with the same covariance. Violations (skewed classes, very different within-class variances) degrade performance, though often gracefully.
• At most C − 1 components. For binary classification, LDA produces only 1 component. For 10 classes, at most 9. If you need more reduced dimensions than C − 1, use PCA or a non-linear method.

Choosing between methods

A practical decision tree:

1. Do you have labels and need class separation? Use LDA (or try PCA if LDA’s C − 1 component limit is too restrictive).
2. Is this for downstream ML (not just visualisation)? Use PCA. It is deterministic, reproducible, invertible, and applies to new data points directly.
3. Is this for visualisation of structure in unlabelled data? Use UMAP (default) or t-SNE (older alternative, less preferred for large datasets).
4. Is your data large (100k+ rows)? UMAP is essentially the only practical non-linear choice; t-SNE becomes prohibitively slow.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
import umap  # pip install umap-learn
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

# X: 10,000 × 128 audio feature matrix (already standardised)
# y: 10,000 genre labels (0=jazz, 1=pop, 2=classical, 3=rock)

# PCA — fast, linear, unsupervised
pca_2d = PCA(n_components=2).fit_transform(X)

# t-SNE — slow, non-linear, unsupervised, preserves local neighbourhoods
tsne_2d = TSNE(n_components=2, perplexity=30, random_state=42).fit_transform(X)

# UMAP — fast, non-linear, unsupervised, preserves local + more global structure
umap_2d = umap.UMAP(n_components=2, n_neighbors=15,
                   min_dist=0.1, random_state=42).fit_transform(X)

# LDA — fast, linear, SUPERVISED, at most C-1 = 3 components for 4 classes
lda_3d = LinearDiscriminantAnalysis(n_components=3).fit_transform(X, y)

Where dimensionality reduction fits overall

After four chapters, the topic’s arc is complete: the curse of dimensionality explained why high-dimensional features are costly, the correlation chapter showed how to identify redundancy, PCA provided the primary tool for replacing correlated features with uncorrelated principal components, and this chapter extended the toolkit for cases where PCA’s linear assumption is too restrictive. In practice, most real dimensionality-reduction work uses PCA (for feature engineering) and UMAP (for visualisation), with LDA for supervised multiclass problems and t-SNE as a legacy tool mostly replaced by UMAP.

The next topic — Data Splits and Evaluation — applies a different discipline to the same concern: instead of reducing features, you measure more honestly. Both are parts of the same underlying goal: build models that generalise, by either simplifying the input space or rigorously testing the output.