The Autoencoder Revisited
An autoencoder is a neural network trained to copy its input to its output through a narrow bottleneck. The squeeze forces it to learn an efficient code.
The two halves
- The encoder maps a high dimensional input to a small latent vector.
- The decoder maps that latent vector back to a reconstruction of the input.
- Training minimizes reconstruction error, usually the squared difference between input and output.
Why the bottleneck helps
- The latent layer is smaller than the input, so the network cannot just copy values. It must keep only the most useful structure.
- The learned code can be used for dimensionality reduction, denoising, or as features for another task.
- A plain autoencoder is not generative. Its latent space has gaps and holes, so sampling a random latent often decodes to nonsense.
This last limitation is exactly what the variational autoencoder was built to fix by giving the latent space a smooth probabilistic structure.
Key idea
An autoencoder learns a compact code by squeezing data through a bottleneck and reconstructing it, but its latent space is not smooth enough to sample new data reliably.