Gradient accumulation is a technique used in deep learning to effectively increase batch size when memory constraints limit the number of samples processed per iteration. Instead of updating model weights after every small batch, gradients are accumulated over several mini-batches before performing a single optimization step. This enables training with larger effective batch sizes without exceeding hardware memory limits.
The process works by summing gradients from multiple forward and backward passes before applying the optimizer. Implementation typically involves zeroing gradients initially, then for each mini-batch within an accumulation step, performing forward and backward passes while accumulating gradients, and finally updating weights after the specified number of iterations. It's crucial to scale the loss appropriately—usually by dividing by the number of accumulation steps—to maintain consistent gradient magnitudes.
Common pitfalls include forgetting to reset gradients after accumulation, improper loss scaling, and mismatched learning rate schedules. Practitioners should ensure that gradient clipping and normalization are applied correctly across accumulated steps. While gradient accumulation can mimic larger batch training, it may introduce subtle differences in training dynamics due to weight updates occurring less frequently.
This technique is widely used in training large models like transformers and GANs, where memory constraints are severe. With careful implementation, gradient accumulation offers a practical workaround for hardware limitations without sacrificing model quality.