Helpful context:

Every model faces the same fundamental problem: it is trained on a finite sample, but judged on the full distribution. The loss you minimize is the empirical risk - an average over the training examples you happened to collect. The loss you actually care about is the expected risk - an average over the true data-generating distribution. These two quantities agree in expectation, but in any given training run, the empirical risk is lower than the expected risk because the model can exploit idiosyncrasies in the training set that do not reflect genuine structure. This gap between empirical and expected risk is the generalization gap, and closing it is the central challenge of machine learning.

Regularization is the collection of techniques that do exactly this. Not by improving the model’s fit to the training data - the optimizer already handles that - but by constraining the hypothesis class, penalizing complexity, or introducing noise, so that the model is forced to learn structure that generalizes rather than structure that only holds for the specific examples it saw. The techniques range from adding a penalty term to the loss, to stopping optimization early, to randomly destroying half the network on every forward pass. They look different on the surface but are unified by a single goal: reduce the gap between training performance and test performance.

This post covers the main regularization methods used in modern deep learning: L2 and L1 weight penalties, early stopping, data augmentation, and dropout. For dropout in particular, we will go beyond the standard “prevents co-adaptation” story and examine the precise connection to ensemble methods - why dropout training approximates training exponentially many models simultaneously, why test-time averaging approximates ensemble averaging, and how Monte Carlo Dropout turns this into a practical tool for uncertainty quantification.

The Overfitting Problem

Let $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ be a training set drawn i.i.d. from some distribution $p(x, y)$. The quantity we care about is the expected risk:

$$R(f) = \mathbb{E}_{(x,y) \sim p}\left[\ell(f(x), y)\right]$$

where $\ell$ is the loss function. Since we cannot compute this expectation directly (we do not have access to $p$), we minimize the empirical risk:

$$\hat{R}(f) = \frac{1}{n}\sum_{i=1}^n \ell(f(x_i), y_i)$$

The empirical risk is an unbiased estimator of the expected risk, meaning $\mathbb{E}_\mathcal{D}[\hat{R}(f)] = R(f)$ for any fixed $f$. The problem is that we do not evaluate at a fixed $f$ - we choose $f$ by minimizing $\hat{R}$, which introduces selection bias. The selected model is the one that best exploits the training data, which is systematically better on the training set than on the full distribution.

The generalization gap is $R(f) - \hat{R}(f)$. Statistical learning theory gives bounds on this gap in terms of the complexity of the hypothesis class. For a class of VC dimension $d$ and $n$ training examples, the gap is bounded by roughly $\mathcal{O}(\sqrt{d/n})$ with high probability. This immediately tells you the two levers: reduce model complexity, or increase data. Regularization is about controlling the first lever - and sometimes approximating the second.

A deep neural network with millions of parameters has astronomically high capacity. Its VC dimension is enormous. Without any constraint, it will find a parameter setting that achieves near-zero training loss by memorizing the training data - fitting even random label noise perfectly. The expected risk can remain high while the empirical risk collapses. This is overfitting in its purest form: low $\hat{R}(f)$, high $R(f)$, large generalization gap.

L2 Regularization (Weight Decay)

The most straightforward fix is to add a penalty for parameter magnitude directly to the loss. L2 regularization augments the training objective with the squared Euclidean norm of the weight vector:

$$\mathcal{L}_{\text{reg}}(w) = \mathcal{L}(w) + \lambda |w|^2 = \mathcal{L}(w) + \lambda \sum_j w_j^2$$

where $\lambda > 0$ is the regularization strength. The gradient becomes:

$$\nabla_w \mathcal{L}_{\text{reg}} = \nabla_w \mathcal{L} + 2\lambda w$$

A gradient descent update with step size $\eta$ then takes the form:

$$w \leftarrow w - \eta(\nabla_w \mathcal{L} + 2\lambda w) = (1 - 2\eta\lambda)w - \eta\nabla_w \mathcal{L}$$

The factor $(1 - 2\eta\lambda)$ multiplies the current weight before the gradient step. This is the “weight decay” interpretation: each update shrinks the weights by a factor slightly less than one, pulling them toward zero, before adding the gradient correction. Large weights are penalized more than small ones. The optimizer is forced to justify large weights by a proportionally large reduction in the data loss.

Bayesian Interpretation

The L2 penalty has a clean probabilistic justification. In Bayesian inference, we compute the posterior over weights given data: $p(w | \mathcal{D}) \propto p(\mathcal{D} | w) \cdot p(w)$. Maximum a posteriori (MAP) estimation maximizes this posterior, or equivalently minimizes the negative log-posterior:

$$-\log p(w | \mathcal{D}) = -\log p(\mathcal{D} | w) - \log p(w) + \text{const}$$

If the likelihood $p(\mathcal{D} | w)$ corresponds to a standard loss $\mathcal{L}(w)$ (as with Gaussian noise for squared loss, or Bernoulli for cross-entropy), and the prior on weights is an isotropic Gaussian:

$$p(w) = \mathcal{N}(0, \sigma_w^2 I) \propto \exp\left(-\frac{|w|^2}{2\sigma_w^2}\right)$$

then the MAP objective is:

$$-\log p(w | \mathcal{D}) = \mathcal{L}(w) + \frac{1}{2\sigma_w^2}|w|^2 + \text{const}$$

This is exactly L2 regularization with $\lambda = \frac{1}{2\sigma_w^2}$. A small $\sigma_w^2$ (tight prior, strong belief that weights should be small) corresponds to large $\lambda$ (heavy regularization). A large $\sigma_w^2$ (diffuse prior) corresponds to small $\lambda$. L2 regularization is not an ad-hoc trick - it is MAP estimation under a Gaussian prior.

Connection to Ridge Regression

In the special case of linear regression with squared loss, L2 regularization has an exact closed-form solution. The objective is:

$$\mathcal{L}_{\text{reg}}(w) = |Xw - y|^2 + \lambda|w|^2$$

Setting the gradient to zero:

$$2X^T(Xw - y) + 2\lambda w = 0 \implies (X^TX + \lambda I)w = X^Ty$$

$$w^* = (X^TX + \lambda I)^{-1}X^Ty$$

This is ridge regression. The matrix $X^TX + \lambda I$ is always invertible (since $\lambda I$ is positive definite), so ridge regression does not suffer from the singularity problems of ordinary least squares when $X^TX$ is ill-conditioned. The regularization shrinks all singular values of $X^TX$ by the same amount $\lambda$, preferentially shrinking the directions corresponding to small singular values - the directions with little data support.

L1 Regularization and Sparsity

L1 regularization replaces the squared norm penalty with the sum of absolute values:

$$\mathcal{L}_{\text{reg}}(w) = \mathcal{L}(w) + \lambda|w|_1 = \mathcal{L}(w) + \lambda \sum_j |w_j|$$

The key behavioral difference from L2 is that L1 regularization produces sparse solutions: many weights become exactly zero at the optimum, rather than being merely small. This is not a numerical coincidence - it is a geometric consequence of the structure of the L1 ball.

Subgradient and Non-differentiability

At $w_j = 0$, the term $\lambda|w_j|$ is not differentiable - the left and right derivatives are $-\lambda$ and $+\lambda$ respectively. We use the subgradient in place of the gradient. The subdifferential of $|w_j|$ at $w_j = 0$ is the interval $[-1, 1]$. An update rule using the subgradient applies:

$$\frac{\partial \mathcal{L}_{\text{reg}}}{\partial w_j} = \frac{\partial \mathcal{L}}{\partial w_j} + \lambda \cdot \text{sign}(w_j)$$

with the convention that $\text{sign}(0) \in [-1, 1]$ is chosen to minimize the subgradient. In practice this is implemented via the soft-thresholding operator: after a gradient step, each weight is shrunk toward zero by $\lambda\eta$, and any weight that would cross zero is set to exactly zero:

$$w_j \leftarrow \text{sign}(w_j - \eta \nabla_{w_j}\mathcal{L}) \cdot \max(|w_j - \eta \nabla_{w_j}\mathcal{L}| - \lambda\eta, 0)$$

This is the proximal gradient step for L1 regularization. The hard zero is how sparsity emerges.

The Geometry of Sparsity

The geometric reason L1 promotes sparsity while L2 does not: consider minimizing a convex loss $\mathcal{L}(w)$ subject to a norm constraint. For L2 the constraint set is a sphere $|w|_2 \leq c$, which is smooth and round. For L1 the constraint set is a diamond (in 2D) or hyperoctahedron (in higher dimensions) $|w|_1 \leq c$, which has sharp corners at the coordinate axes. The unconstrained loss minimum is usually not inside the constraint set. The constrained solution sits at the boundary where the level set of $\mathcal{L}(w)$ first touches the constraint set. For the smooth L2 ball, this touching point is generic and rarely lands exactly on an axis. For the L1 diamond, the corners (coordinate axes) are the most likely touching points because they are the “pointiest” parts of the boundary - level sets of smooth functions tend to hit corners rather than flat faces. A corner on a coordinate axis means some $w_j = 0$ exactly.

In high dimensions with $p$ features, L1 regularization (also called Lasso in the regression context) finds solutions with at most $n$ nonzero coefficients when $n < p$. This makes it valuable for feature selection: the nonzero weights identify which inputs actually matter.

Elastic Net

Elastic Net combines L1 and L2 penalties:

$$\mathcal{L}_{\text{reg}}(w) = \mathcal{L}(w) + \lambda_1|w|_1 + \lambda_2|w|^2$$

The L2 term stabilizes the optimization when features are correlated (Lasso tends to pick one arbitrarily from a correlated group; Elastic Net can select all of them with shared weight). The L1 term maintains sparsity. In practice, Elastic Net often outperforms either penalty alone when $p \gg n$ and features are correlated.

Early Stopping

Early stopping is a regularization technique that requires no modification to the loss function. The idea: validation loss typically decreases initially as training progresses, then starts to increase as the model begins overfitting to the training data. Stop training when the validation loss stops improving.

The practical procedure: keep a copy of the best model weights seen so far (according to validation loss). After each epoch (or every $k$ steps), evaluate on the validation set. If validation loss improves, update the saved weights. If validation loss has not improved for $T$ consecutive evaluations (the “patience” parameter), stop and return the saved weights.

Why does this work as regularization? Gradient descent from a random initialization traces a trajectory through parameter space. Early in training, the trajectory moves toward the “center” of the parameter space - the region where loss is low for many possible training sets, not just this one. Later in training, it starts exploiting the specific training data, moving into regions that have low training loss but high test loss. Early stopping limits how far along this trajectory you travel. It is a form of implicit regularization - the optimization trajectory itself is being constrained, not the loss function.

There is a formal connection to L2 regularization for quadratic losses. For a quadratic loss near the minimum, early stopping with gradient descent is approximately equivalent to L2 regularization, with the regularization strength determined by the stopping time: shorter training time $\approx$ stronger regularization. This gives an intuition for why both methods have similar effects, even though their mechanisms look different.

Data Augmentation

If overfitting occurs because the model exploits finite-sample artifacts, the most direct fix is to increase the effective size of the training set. Data augmentation creates additional training examples by applying label-preserving transformations to existing examples.

For images, common augmentations include: random crops and resizes, horizontal flips, rotations, color jitter (brightness, contrast, saturation shifts), cutout (masking random rectangular regions), and mixup (interpolating two images and their labels: $\tilde{x} = \alpha x_i + (1-\alpha)x_j$, $\tilde{y} = \alpha y_i + (1-\alpha)y_j$). For text: synonym replacement, back-translation (translate to another language and back), random deletion of words, word order perturbations. For audio: time stretching, pitch shifting, noise injection, SpecAugment (masking blocks of time-frequency bins).

The regularization effect is genuine, not just effective sample size increase. Augmentation injects inductive biases: using random flips tells the model that horizontal orientation should not affect the prediction. Using random crops tells the model that the object can appear anywhere. The model must be invariant to these transformations to minimize the augmented training loss, which is exactly what we want for generalization.

Noise injection as augmentation has a direct connection to regularization: adding Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ to inputs during training is equivalent to adding a term $\frac{\sigma^2}{2}\mathbb{E}\left[|\nabla_x \mathcal{L}|^2\right]$ to the loss, which penalizes large input gradients and promotes smooth decision boundaries. This is a form of Tikhonov regularization in input space.

Dropout

Dropout is a regularization technique specific to neural networks, introduced by Srivastava et al. in 2014. During each forward pass in training, every neuron’s activation is independently set to zero with probability $p$ (the dropout probability, or “drop rate”). Neurons that are kept have their activations scaled by $\frac{1}{1-p}$. At test time, all neurons are used and no scaling is applied.

More precisely, if $h$ is the vector of pre-scaling activations at some layer, the dropout mask is $m \sim \text{Bernoulli}(1-p)^d$ (i.i.d. for each unit), and the masked activation is:

$$\tilde{h} = \frac{m \odot h}{1-p}$$

The scaling by $\frac{1}{1-p}$ ensures that $\mathbb{E}[\tilde{h}] = h$, so the expected activation is unchanged. This is inverted dropout - the scaling happens during training, so test time requires no modification to the network. The alternative (scale at test time by $1-p$) produces identical expected outputs but requires remembering to apply the scaling at inference, which is error-prone.

The dropout probability $p$ is a hyperparameter. Common choices are $p = 0.5$ for fully connected layers and $p = 0.1$ to $0.2$ for convolutional layers (which are already regularized by weight sharing). Dropout is typically not applied to batch normalization layers or the output layer.

Dropout as Approximate Bagging

The most illuminating way to understand why dropout works is the ensemble / bagging interpretation, due to Hinton et al. and formalized by Goodfellow et al.

Bagging (bootstrap aggregating) is a general ensemble method: train $K$ separate models on $K$ bootstrap samples of the training data, then average their predictions. It reduces variance without increasing bias, because averaging reduces the variance of a sum of identically distributed random variables by a factor of $K$.

Dropout implements a form of bagging that is computationally much cheaper. Here is the precise connection.

A network with $n$ neurons subject to dropout implicitly defines $2^n$ different network architectures - one for each possible dropout mask $m \in \{0,1\}^n$. Each distinct mask $m$ defines a “thinned” sub-network. During training, each mini-batch uses a freshly sampled mask, so effectively trains a different sub-network. The weights are shared across all sub-networks: the same weight $w_{ij}$ appears in every sub-network that includes both neuron $i$ and neuron $j$. This weight sharing means that each sub-network is not trained from scratch - it benefits from the training done by all the other sub-networks that share its weights.

The training procedure is equivalent to: sample a sub-network $m$ from the uniform distribution over masks, train it for one step using the current mini-batch, then do the same for the next mini-batch with a new sample. Over the course of training, each possible sub-network is updated roughly proportionally to how often it is sampled. With $2^n$ possible masks and $T$ training steps, most masks are sampled at most once or twice - this is genuine diversity across sub-networks, analogous to training separate models in a bagging ensemble.

At test time, we want to compute the ensemble prediction:

$$\hat{y} = \frac{1}{2^n}\sum_{m \in \{0,1\}^n} f(x; w \odot m)$$

Summing over all $2^n$ masks is intractable for any realistic network. The key insight: the geometric mean of the ensemble predictions is approximately computed by the single full network with weights scaled by $1-p$. Dropout at test time is replaced by the “weight scaling inference rule” - use all weights, scaled by $(1-p)$ (or equivalently, use all activations scaled by $(1-p)$). This is exact for a single sigmoid output unit (because the geometric mean of Bernoulli mixtures satisfies a product form), and is a good approximation for deeper networks.

The approximation is justified because: (1) the arithmetic mean of predictions is well-approximated by the geometric mean for sharply peaked predictive distributions; (2) empirically, the weight scaling rule matches Monte Carlo approximations of the true ensemble average closely; and (3) the scaling rule is exact for linear networks, and deep networks with smooth activations are locally approximately linear in many directions.

Why Approximate Bagging Helps

The variance reduction from bagging is $\text{Var}\left(\frac{1}{K}\sum_k f_k\right) = \frac{1}{K}\text{Var}(f) + \frac{K-1}{K}\text{Cov}(f_i, f_j)$. With $K = 2^n$ sub-networks, the $\frac{1}{K}$ factor is negligible, but the covariance term matters. Sub-networks trained with shared weights are more correlated than independently trained models (the standard bagging setting). So dropout provides variance reduction, but less than ideal bagging. This is the “approximate” in approximate bagging - it is a practical compromise between the full variance reduction of true bagging and the computational cost of training a single network.

Connection to Bayesian Model Averaging

The bagging interpretation connects dropout to Bayesian model averaging (BMA). In BMA, the predictive distribution is:

$$p(y | x, \mathcal{D}) = \int p(y | x, w) p(w | \mathcal{D}) dw$$

This integral averages predictions over all weight settings, weighted by their posterior probability. The ensemble of sub-networks in dropout approximates this integral, with the mask distribution playing the role of the posterior over model architectures. Each mask selects a different model, and averaging over masks approximates averaging over models. The connection is not exact - masks are sampled uniformly, not from a posterior - but it motivates the Bayesian interpretation of dropout and connects to principled uncertainty quantification, which we examine next.

Monte Carlo Dropout

The ensemble interpretation of dropout suggests a natural extension: instead of using the deterministic weight-scaling inference rule at test time, keep dropout active and run multiple stochastic forward passes. This is Monte Carlo Dropout, proposed by Gal and Ghahramani (2016).

The procedure: given a test input $x$, run $T$ stochastic forward passes through the network with dropout active. Each pass uses a freshly sampled mask $m^{(t)}$, producing a prediction $\hat{y}^{(t)} = f(x; w \odot m^{(t)})$. The predictive mean and variance are estimated as:

$$\bar{y} = \frac{1}{T}\sum_{t=1}^T \hat{y}^{(t)}$$

$$\text{Var}(y | x) \approx \frac{1}{T}\sum_{t=1}^T \hat{y}^{(t)2} - \bar{y}^2 + \frac{1}{T}\sum_{t=1}^T \hat{\sigma}^{(t)2}$$

where the last term accounts for aleatoric uncertainty if the model also outputs a predicted noise variance $\hat{\sigma}^2$. The first two terms capture epistemic uncertainty - uncertainty due to limited data, which should decrease as more training data is collected. The last term captures aleatoric uncertainty - irreducible noise in the labels.

Formal Justification

Gal and Ghahramani showed that a dropout neural network with Gaussian priors on the weights is mathematically equivalent to a deep Gaussian process approximated by variational inference. The dropout masks correspond to variational parameters in a Bernoulli approximating distribution, and training the network minimizes a variational lower bound on the marginal likelihood. Test-time dropout with Monte Carlo averaging is then exactly the variational inference procedure for the posterior predictive distribution.

More concretely: the variational distribution over weights $q(w)$ is a product of Bernoulli distributions (one per weight), parameterized by the trained weights and the dropout rate. Monte Carlo Dropout samples from this variational distribution and averages predictions, which is the standard MC approximation to the posterior predictive:

$$p(y | x, \mathcal{D}) \approx \frac{1}{T}\sum_{t=1}^T p(y | x, w^{(t)}), \quad w^{(t)} \sim q(w)$$

Practical Use in Uncertainty Quantification

MC Dropout gives you uncertainty estimates from a model you have already trained with standard dropout - no modification to the architecture or training procedure is required. This is its main practical advantage over other Bayesian deep learning approaches (Laplace approximations, deep ensembles, variational Bayes), which require either retraining or more complex model changes.

In practice, $T = 30$ to $100$ forward passes is typically sufficient. The variance across passes gives a reliable signal for uncertainty: inputs where the network is uncertain (out-of-distribution, ambiguous) tend to produce high variance across passes, while in-distribution inputs produce low variance. This has been applied to active learning (query the examples with highest uncertainty), anomaly detection (flag high-uncertainty inputs), and safety-critical systems (abstain from prediction when uncertainty is high).

One limitation: the uncertainty estimates are only as good as the variational approximation. Bernoulli variational distributions are not expressive enough to capture complex posterior shapes over weights, and the approximation can underestimate uncertainty in some settings. Deep ensembles (training multiple independent networks from different random initializations) often provide better-calibrated uncertainty at the cost of higher computational budget.

Summary

Technique Mechanism Inductive Bias When to Use
L2 (weight decay) Add $\lambda|w|^2$ to loss; weight decay update $(1-2\eta\lambda)w - \eta\nabla\mathcal{L}$ Gaussian prior on weights; small weights preferred Almost always; default choice for neural networks
L1 (Lasso) Add $\lambda|w|_1$ to loss; soft-thresholding updates Laplace prior; sparse solutions Feature selection; high-dimensional, low-signal settings
Elastic Net L1 + L2 combined Sparsity + stability for correlated features Correlated features, high-dimensional regression
Early stopping Stop when validation loss increases; return best checkpoint Limits trajectory length in parameter space Always; easy to implement alongside any training run
Data augmentation Label-preserving input transformations Invariances encoded by the transform family Images, audio, text; transformations must preserve labels
Dropout Random mask $m \sim \text{Bernoulli}(1-p)^n$ per forward pass; scale by $1/(1-p)$ Approximate ensemble of $2^n$ sub-networks Fully connected layers; less useful with batch norm
MC Dropout Keep dropout at test time; $T$ forward passes for mean/variance Variational Bayes over weights Uncertainty quantification; no extra training cost

All regularization techniques address the same root problem - the generalization gap between empirical and expected risk - but from different angles. L2 and L1 constrain the hypothesis class directly through the loss. Early stopping constrains the optimization trajectory. Data augmentation expands the effective training distribution. Dropout trains an implicit ensemble with shared weights and provides an approximation to Bayesian model averaging. In practice, the best results come from combining multiple techniques: weight decay plus dropout plus data augmentation is the standard recipe for image classification, and it works because the variance reductions from different methods compound.

Read next: