[Paper Review] Adding Gradient Noise Improves Learning for Very Deep Networks

Arvind Neelakantan, Luke Vilnis+5 more|arXiv (Cornell University)|Nov 21, 2015

Domain Adaptation and Few-Shot Learning31 references265 citations

TL;DR

This paper proposes adding annealed Gaussian gradient noise during stochastic gradient descent to improve training of very deep neural networks. The method enhances optimization by encouraging exploration of parameter space, enabling successful training of a 20-layer fully connected network from poor initialization and achieving a 72% relative error reduction on a question-answering task and doubling the number of accurate binary multiplication models across 7,000 random restarts.

ABSTRACT

Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks. The main motivation for these architectural innovations is that they capture better domain knowledge, and importantly are easier to optimize than more basic architectures. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we discuss a low-overhead and easy-to-implement technique of adding gradient noise which we find to be surprisingly effective when training these very deep architectures. The technique not only helps to avoid overfitting, but also can result in lower training loss. This method alone allows a fully-connected 20-layer deep network to be trained with standard gradient descent, even starting from a poor initialization. We see consistent improvements for many complex models, including a 72% relative reduction in error rate over a carefully-tuned baseline on a challenging question-answering task, and a doubling of the number of accurate binary multiplication models learned across 7,000 random restarts. We encourage further application of this technique to additional complex modern architectures.

Motivation & Objective

• Address optimization challenges in very deep and complex neural architectures such as Neural Turing Machines and Memory Networks.
• Overcome difficulties in training deep feedforward and recurrent networks, especially when initialized poorly.
• Improve generalization and robustness across random initializations and hyperparameter settings.
• Explore a low-overhead, easy-to-implement technique that enhances training without architectural changes.
• Demonstrate consistent performance gains across diverse complex models, including those for algorithm learning and question answering.

Proposed method

• Add zero-mean Gaussian noise to the gradient after gradient clipping during backpropagation.
• Use an annealing schedule for the noise variance, decreasing it over time according to a predefined decay function.
• Apply the noise to the gradient in the standard stochastic gradient descent update step.
• Maintain the same optimization hyperparameters (e.g., learning rate, batch size) as baseline models.
• Implement the noise injection with a single line of code, making it highly practical and deployable.
• Use Adam optimizer in experiments, with noise applied after gradient clipping to stabilize updates.

Experimental results

Research questions

• RQ1 Can adding gradient noise improve training stability and convergence in very deep feedforward and recurrent networks?
• RQ2 Does gradient noise enhance generalization and robustness across multiple random initializations?
• RQ3 Can gradient noise help train deep networks from poor initializations where standard SGD fails?
• RQ4 How does gradient noise affect performance on complex tasks such as question answering and algorithm learning?
• RQ5 Does the annealing schedule of gradient noise provide measurable improvements over constant or non-annealed noise?

Key findings

• The addition of annealed gradient noise enabled successful training of a 20-layer rectified fully-connected network on MNIST using standard stochastic gradient descent, even from a poor initialization.
• On a challenging question-answering task, the method achieved a 72% relative reduction in error rate compared to a carefully-tuned baseline.
• In a large-scale experiment with 7,290 random restarts on binary multiplication, models trained with gradient noise achieved over twice as many accurate results (error < 1%) compared to the baseline without noise.
• The method improved robustness across hyperparameter settings and initializations, with 11.3% of runs succeeding with noise versus 1.3% without on the k-th element task.
• Gradient noise reduced training loss and improved generalization, suggesting it aids in escaping poor local minima in complex loss landscapes.
• The technique consistently improved performance across diverse architectures, including fully-connected networks, Neural GPUs, and question-answering models.