An autoencoder is a neural network trained to copy its input to its output — but with a twist. It must pass through a bottleneck (latent space) that is smaller than the input. The encoder compresses the input into a compact latent representation z, and the decoder reconstructs the original input from z. The network is trained by minimizing reconstruction loss (typically MSE between input and output). The bottleneck forces the network to learn the most important features of the data.

The latent space z is a low-dimensional representation of the input. For face images, latent dimensions might correspond to features like pose, lighting, expression, or hair color — even though nobody told the network about these concepts. An undercomplete autoencoder (latent dim < input dim) is forced to learn a compressed representation. An overcomplete autoencoder (latent dim ≥ input dim) can trivially copy the input, so it needs regularization (sparsity, noise) to learn useful features.

A denoising autoencoder corrupts the input (adding Gaussian noise, masking random pixels) and trains the network to reconstruct the clean original. This prevents the network from learning the identity function and forces it to capture robust features. The corruption acts as regularization, making the learned representations more generalizable. This idea directly inspired BERT’s masked language modeling — corrupting input tokens and predicting the originals.

A sparse autoencoder adds a penalty that encourages most latent units to be inactive (near zero) for any given input. Only a few units “fire” for each input, creating a distributed, sparse code. This is inspired by neuroscience — only a small fraction of neurons in the brain are active at any time. Sparsity is enforced via L1 regularization or a KL divergence penalty on the average activation.

A standard autoencoder maps inputs to specific points in latent space. But what if you want to generate new data? You’d need to sample from the latent space, but the space has no structure — there are gaps and irregular regions. The Variational Autoencoder (VAE) (Kingma & Welling, 2014) solves this by making the encoder output a probability distribution (mean μ and variance σ²) instead of a single point. The latent code z is sampled from this distribution. A KL divergence loss regularizes the distribution to be close to a standard normal N(0, 1), ensuring the latent space is smooth and continuous.

A disentangled representation is one where each latent dimension corresponds to a single, independent factor of variation. For face images: dimension 1 controls pose, dimension 2 controls lighting, dimension 3 controls expression — and changing one doesn’t affect the others. β-VAE (Higgins et al., 2017) achieves this by increasing the weight of the KL divergence term (β > 1), which pressures each latent dimension to be more independent.

For image data, the encoder uses convolutional layers with stride-2 to progressively downsample, and the decoder uses transposed convolutions (or upsampling + convolution) to upsample back to the original resolution. This preserves spatial structure far better than flattening images into vectors. Modern convolutional VAEs can generate realistic faces, rooms, and objects at resolutions up to 256×256.

VAEs learn an explicit latent space with smooth interpolation but produce blurry outputs (due to the MSE loss averaging over possibilities). GANs (next chapter) produce sharp images but have no encoder and are hard to train. Diffusion models (2020+) learn to denoise, producing the highest-quality images but requiring many sampling steps. Each has trade-offs between sample quality, training stability, latent space structure, and inference speed.

Autoencoders learn compressed representations by reconstructing their input through a bottleneck. VAEs add probabilistic structure to the latent space, enabling generation of new data. Disentangled representations separate independent factors of variation. These ideas — latent spaces, encoders, decoders, and the reparameterization trick — are foundational to modern generative AI, from Stable Diffusion’s latent space to the tokenizers used in vision transformers.