Overfitting occurs when a model performs well on training data but poorly on unseen data — it has memorized the training examples rather than learning generalizable patterns. Deep networks with millions of parameters are especially prone to overfitting because they have enough capacity to memorize entire datasets. The bias-variance tradeoff: high bias (underfitting) means the model is too simple; high variance (overfitting) means it’s too complex. Regularization techniques reduce variance without significantly increasing bias.

During training, dropout randomly sets each neuron’s output to zero with probability p (typically 0.5 for FC layers, 0.1–0.3 for conv layers). This prevents neurons from co-adapting — no neuron can rely on any specific other neuron being present. It’s like training an ensemble of 2ⁿ different sub-networks (where n is the number of neurons). At inference time, all neurons are active but outputs are scaled by (1-p) to compensate. PyTorch handles this automatically with nn.Dropout.

Batch Normalization normalizes each layer’s inputs to have zero mean and unit variance across the mini-batch, then applies learnable scale (γ) and shift (β) parameters. This addresses internal covariate shift — the distribution of layer inputs changing as earlier layers update. BatchNorm allows higher learning rates, reduces sensitivity to initialization, and acts as a mild regularizer. It was the single most impactful technique for training deep CNNs and is used in virtually every modern architecture.

Layer Norm normalizes across all features for each sample independently — no batch dependency. Standard for transformers (GPT, BERT, LLaMA). Group Norm (Wu & He, 2018) divides channels into groups and normalizes within each group — works well for small batches in detection/segmentation. RMSNorm (Zhang & Sennrich, 2019) simplifies LayerNorm by removing the mean centering, using only root-mean-square normalization. LLaMA and many modern LLMs use RMSNorm for efficiency.

Data augmentation applies random transformations to training images (flips, rotations, crops, color jitter) to artificially increase dataset size and diversity. A dataset of 10K images with augmentation can behave like 100K+ images. This is one of the most effective regularization techniques — it directly addresses the root cause of overfitting (insufficient data diversity). Modern augmentation strategies like RandAugment (Cubuk et al., 2020) and Mixup (Zhang et al., 2018) have become standard.

Early stopping monitors validation loss during training and stops when it starts increasing (while training loss continues decreasing). This is the simplest and most effective way to prevent overfitting. Typically, you save the model checkpoint with the best validation loss and use patience (wait N epochs after the best score before stopping) to avoid stopping too early due to noise.

Weight decay adds a penalty proportional to the squared magnitude of weights: L_total = L_data + λ||w||². This discourages large weights, pushing the model toward simpler solutions. In AdamW, weight decay is decoupled from the gradient update (λ typically 0.01–0.1). Weight decay is used in virtually every deep learning training recipe.

Mixup creates new training examples by linearly interpolating between pairs of images and their labels: x_new = λ·x₁ + (1-λ)·x₂, y_new = λ·y₁ + (1-λ)·y₂. This encourages the model to behave linearly between training examples, producing smoother decision boundaries. CutMix (Yun et al., 2019) cuts a rectangular patch from one image and pastes it onto another, mixing labels proportionally to the area. Both significantly improve generalization on ImageNet.

Label smoothing (Szegedy et al., 2016) replaces hard labels [0, 0, 1, 0] with soft labels [0.033, 0.033, 0.9, 0.033]. Instead of pushing the model to be 100% confident, it encourages calibrated uncertainty. This prevents the model from becoming overconfident and improves generalization. Label smoothing of 0.1 is standard for ImageNet training and transformer models.

Modern training recipes combine multiple techniques. For ImageNet CNNs: AdamW or SGD+momentum, cosine LR schedule, RandAugment, Mixup/CutMix, label smoothing 0.1, weight decay 0.05, and stochastic depth. For transformers/LLMs: AdamW (lr=3e-4, wd=0.1), linear warmup + cosine decay, gradient clipping, RMSNorm or LayerNorm, and dropout 0.1. These recipes have been refined through thousands of experiments and represent hard-won practical knowledge.