What PCA
can’t see.
A pattern can be right in front of us.
And still be invisible to the way we look.
Three small experiments on nonlinear autoencoders, hidden structure, and what it means to learn something useful.
Follow the ideaThe dots follow a curve, even though their linear correlation is zero.
Uncorrelated.
Still connected.
PCA looks for directions with the most variance. But covariance only captures part of the picture. Here, the coordinates have equal variance and zero covariance: PCA has no preferred direction, while the curve is easy to see.
Try the nonlinear lens. We supply the feature x², which exposes the relationship. This is a toy illustration, not a trained model; squaring x also loses its sign.
How does this connect to the paper?
The paper hides two directions in high-dimensional noise. One appears in covariance; the other is whitened so its extra variance disappears. Their latent factors remain dependent beyond covariance. Our two-dimensional sketch illustrates the blind spot, not that full model. Keeping both PCA components would still retain this entire toy dataset. Read §1 ↗
Less room.
What survives?
An autoencoder compresses an input into a small code, then tries to reconstruct it. That code is its representation.
Make the bottleneck smaller. Watch which details make it through.
64 pixel values
the bottleneck
an approximation of the input
A good copy is only one kind of success.
Reconstruction error tells us how much of the input was reproduced. It doesn’t tell us whether the code captured a factor we need for another task.
What this experiment is doing
This is a fixed linear encoder and decoder: it keeps a chosen number of low-frequency image components using a discrete cosine transform. No network is trained here. It illustrates compression and reconstruction; the paper studies a different, learned nonlinear autoencoder.
Better at what?
Two models. Two ways to judge them.
Change the question and the verdict changes.
The linear one.
It has lower reconstruction loss on new samples. For making a faithful copy, it comes out ahead.
Linear autoencoder
the PCA baselineNonlinear autoencoder
with a suitable activationThe nonlinear model partly recovers the hidden direction and performs better on the paper’s downstream classification task. No numerical scores are implied. §6, Figure 2 ↗
Learning to reconstruct
isn’t the same as learning
what matters.
A small model.
A useful distinction.
Under the paper’s conditions, even a single hidden unit with a suitable nonlinearity can learn a direction correlated with both planted directions. This is partial recovery, not perfect disentanglement, and it is not a universal claim that nonlinear models beat PCA.
A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalization
Vicente Conde Mendes, Lorenzo Bardone, Cédric Koller, Jorge Medina Moreira, Vittorio Erba, Emanuele Troiani & Lenka Zdeborová.