Autoencoders for Dimensionality
An autoencoder is a neural network that learns to compress and rebuild its own input. The compression forces it to discover a compact representation, making it a learned form of dimensionality reduction.
Encoder bottleneck decoder
An autoencoder has three parts.
- The encoder maps the input down to a small vector called the latent code.
- The bottleneck is that small layer, which is narrower than the input.
- The decoder expands the code back toward the original input.
Because the bottleneck is small, the network must keep only the most useful information to reconstruct the data well.
Training objective
The network is trained to minimize reconstruction error, usually the squared difference between the input and the output. No labels are needed, so it is unsupervised.
Why it beats linear methods
A linear autoencoder with squared loss recovers the same subspace as PCA. With nonlinear activations and multiple layers, an autoencoder can capture curved manifolds that PCA cannot, learning richer low dimensional codes. The trade off is that it needs more data and tuning, and it can overfit if the bottleneck is too wide.
Key idea
An autoencoder compresses input through a bottleneck and reconstructs it, learning nonlinear low dimensional codes that generalize PCA.