Skip to content

Learn / Unsupervised

Dimensionality reduction

Squeezing many features into a few while keeping what matters, for plotting, compression and cleaner inputs.

Jump to the playground

On this page
  1. Intuition
  2. What the data looks like
  3. How the data is prepared
  4. Algorithms
  5. The maths
  6. Real-world examples
  7. Playground
  8. In Python
  9. Pitfalls

Intuition

A hundred features rarely carry a hundred independent pieces of information. Dimensionality reduction finds a smaller set of coordinates that preserves most of the structure. Linear methods (PCA) rotate the data and drop the least informative axes. Non-linear methods (t-SNE, UMAP) try to keep neighbours together, even if that means bending space.

What the data looks like

Each row has many columns. The method keeps the rows and replaces the columns with a few new ones.

Spending profile: many columns per customer, squeezed to two One row = one customer with 50 spending-category columns. Answer: none in the data; the method outputs 2 new coordinates.
rowcustomerinputfoodinputfuelinputtravelinputonlineinput…outputpc1outputpc2
C-30010.310.120.020.2…1.84-0.22
C-30020.080.020.410.33…-2.11.35
C-30030.290.150.010.18…1.62-0.4
C-30040.10.010.380.36…-1.951.21

… plus 44 more spending columns.

Fifty columns per row go in; two coordinates per row come out, ready to plot or to feed another model.

Example rows are invented for illustration.

How the data is prepared

  • Centre and scale. PCA follows variance, so an unscaled feature in large units becomes the first component by default.
  • Remove or cap outliers; a few of them can define a component.
  • For t-SNE and UMAP, reduce to 30-50 dimensions with PCA first. It is faster and removes noise.

Algorithms

PCA

Rotate the data onto the directions of greatest variance and keep the first few.

Cost
O(n·d²)
Strength
Fast, deterministic, invertible.
Weakness
Linear: it cannot unroll curved structure.
Used for
compression, decorrelating features
Open in the playground

Kernel PCA

PCA in an implicit non-linear feature space.

Cost
O(n³)
Strength
Captures some non-linear structure.
Weakness
n×n kernel matrix; kernel choice matters.
Used for
non-linear denoising
Open in the playground

t-SNE

Place points in 2-D so that neighbours stay neighbours.

Cost
O(n²) exact
Strength
Excellent local cluster separation.
Weakness
Distances between clusters mean little; not reusable on new data.
Used for
visualising embeddings
Open in the playground

UMAP

Build a neighbour graph and lay it out in low dimensions.

Cost
O(n log n)
Strength
Fast; keeps more global structure than t-SNE.
Weakness
Stochastic; hyper-parameters change the picture.
Used for
embedding maps, single-cell data
Open in the playground

The maths

PCA finds the unit direction ww along which the centred data varies most:

w1=arg⁡max⁡∥w∥=1  w⊤Σ w,Σ=1nX⊤Xw_1 = \arg\max_{\lVert w \rVert = 1} \; w^\top \Sigma\, w, \qquad \Sigma = \frac{1}{n} X^\top X

The solutions are the eigenvectors of the covariance matrix Σ\Sigma, and each eigenvalue λj\lambda_j is the variance captured by that component. The share kept by the first mm components is ∑j≤mλj/∑jλj\sum_{j \le m} \lambda_j / \sum_j \lambda_j.

t-SNE converts distances into neighbour probabilities pijp_{ij} in the original space and qijq_{ij} in the map, then moves the map points to minimise the Kullback-Leibler divergence

KL(P ∥ Q)=∑i≠jpijlog⁡pijqij\mathrm{KL}(P \,\Vert\, Q) = \sum_{i \ne j} p_{ij} \log \frac{p_{ij}}{q_{ij}}

Perplexity sets roughly how many neighbours each point cares about.

Real-world examples

  • Visualising embeddings: seeing whether support tickets or documents form topics.
  • Compression and denoising: keeping the top components and discarding the rest.
  • Decorrelating features before a model that struggles with collinearity.

Playground

The swiss roll is a flat sheet rolled up in 3-D. PCA can only look at it from one angle, so the layers stay on top of each other. t-SNE and UMAP follow the sheet and unroll it. Drag the 3-D view to see the input.

Data
More data options
Model A
Model B

Uses at most 1500 training points.

Input · first 3 of 3 dimensions

Drag the 3-D view to rotate it.

A · PCA

Drag the 3-D view to rotate it.

B · UMAP

Press Train to compare.

loss per step (axis fitted to the range)

Train a model to see its loss curve.

Step -train ━test ┅

colour = position along the manifold

Press Train. The verdict appears here.

In Python

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import umap

Xs = StandardScaler().fit_transform(X)

pca = PCA(n_components=0.95).fit(Xs)           # keep 95% of the variance
print(pca.n_components_, pca.explained_variance_ratio_[:5])

X50 = PCA(n_components=50).fit_transform(Xs)   # denoise first...
X2 = umap.UMAP(n_neighbors=15, min_dist=0.1).fit_transform(X50)  # ...then map to 2-D for plotting

Pitfalls

  • Reading distances between clusters in t-SNE or UMAP plots. Only local neighbourhoods are meaningful.
  • Cluster sizes in those plots are not meaningful either.
  • One run, one perplexity. Change the settings and the seed before believing a pattern.
  • Using a t-SNE map as model features. It cannot place new points; use PCA or UMAP’s transform.