Dropout Regularization

Dropout is the most widely used regularization technique for neural networks. During training, it randomly deactivates neurons — forcing the network to learn redundant representations that don’t rely on any single path through the network. The result: substantially better generalization with almost no computational cost.

How Dropout Works

During each forward pass, every neuron has probability p of being “dropped” (set to zero):

Without dropout:
  Layer output: [0.8, 0.3, 0.9, 0.2, 0.7]

With dropout (p=0.5):
  Mask:         [  1,   0,   1,   0,   1]  (random each step)
  After drop:   [0.8, 0.0, 0.9, 0.0, 0.7]
  After scale:  [1.6, 0.0, 1.8, 0.0, 1.4]  (scaled by 1/(1-p) to maintain expected sum)

The scaling ensures that the expected output magnitude is the same whether dropout is on or off — critical for inference.

Training vs. Inference

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(512, 256),
    nn.ReLU(),
    nn.Dropout(p=0.5),    # Drops 50% of neurons during training
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.3),    # Lighter dropout near output
    nn.Linear(128, 10)
)

# Training mode: dropout is ACTIVE
model.train()
output = model(x)

# Inference mode: dropout is DISABLED (all neurons active, scaled)
model.eval()
with torch.no_grad():
    predictions = model(x)

Always call model.eval() before inference and model.train() before training. Forgetting this is one of the most common bugs in PyTorch code.

Choosing the Dropout Rate

Spatial Dropout (for CNNs)

Standard dropout applied to feature maps drops individual pixels. Spatial dropout drops entire feature map channels — more effective for convolutional layers:

# For 2D feature maps (batch, channels, H, W)
spatial_dropout = nn.Dropout2d(p=0.2)

# For 1D sequences (batch, channels, length)
spatial_dropout_1d = nn.Dropout1d(p=0.2)

MC Dropout: Uncertainty Estimation

By keeping dropout active at inference time and running multiple forward passes, you can estimate prediction uncertainty — useful for safety-critical applications:

def mc_dropout_predict(model, x, n_samples=50):
    model.train()  # Keep dropout active!
    predictions = []

    with torch.no_grad():
        for _ in range(n_samples):
            pred = model(x)
            predictions.append(torch.softmax(pred, dim=1))

    predictions = torch.stack(predictions)
    mean_pred = predictions.mean(dim=0)
    uncertainty = predictions.std(dim=0)  # High std = uncertain prediction

    return mean_pred, uncertainty

mean, uncertainty = mc_dropout_predict(model, x_test)

Dropout vs. Other Regularization

These techniques are complementary — modern networks typically combine dropout + L2 weight decay + batch normalization.

Dropout in Transformers

In Transformer architectures, dropout is applied in several places:

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, ff_dim, dropout=0.1):
        super().__init__()
        self.attention_dropout = nn.Dropout(dropout)  # After attention weights
        self.ff_dropout = nn.Dropout(dropout)          # After feedforward
        self.residual_dropout = nn.Dropout(dropout)    # After residual additions

A dropout rate of 0.1 is standard in most Transformer implementations.

When Dropout Hurts

Dropout can slow convergence and sometimes hurts performance when:

• The model is already small/simple (less overfitting risk)
• The dataset is very large (data already provides regularization)
• Batch normalization is used heavily (they interact poorly in some architectures)
• Recurrent networks (LSTM) — use variational dropout instead of standard dropout

In practice, always run a quick experiment with and without dropout to verify it helps.