[Paper Review] The Effect of Gradient Noise on the Energy Landscape of Deep Networks

Pratik Chaudhari, Stefano Soatto|arXiv (Cornell University)|Nov 20, 2015

Model Reduction and Neural Networks41 references3 citations

TL;DR

This paper proposes a novel annealing scheme for additive gradient noise in deep learning by modeling the training process as a spherical spin glass under an external magnetic field. By leveraging topology trivialization—where increasing noise magnitude reduces critical points from exponential to one—the method accelerates training and improves generalization, validated on fully-connected and convolutional networks.

ABSTRACT

We analyze the regularization properties of additive gradient noise in the training of deep networks by posing it as finding the ground state of the Hamiltonian of a spherical spin glass in an external magnetic field. We show that depending upon the magnitude of the magnetic field, the Hamiltonian changes dramatically from a highly non-convex energy landscape with exponentially many critical points to a regime with polynomially many critical points and finally, trivializes' to exactly one minimum. This phenomenon, known as topology trivialization in the physics literature, can be leveraged to devise annealing schemes for additive noise such that the training starts in the polynomial regime but gradually morphs the energy landscape into the original one as training progresses. We demonstrate through experiments on fully-connected and convolutional neural networks that annealing schemes based on trivialization lead to accelerated training and also improve generalization error.

Motivation & Objective

• To understand the regularization effects of additive gradient noise in deep network training.
• To model the energy landscape of deep networks under gradient noise using statistical physics concepts.
• To identify how noise magnitude alters the topology of the loss landscape, particularly the number of critical points.
• To develop an annealing scheme that exploits topology trivialization for improved training efficiency and generalization.
• To empirically validate the proposed method on fully-connected and convolutional neural networks.

Proposed method

• The training process is modeled as finding the ground state of a spherical spin glass Hamiltonian under an external magnetic field.
• The magnetic field strength corresponds to the magnitude of additive gradient noise.
• The system's phase transitions are analyzed to identify regimes with exponentially many critical points, polynomially many, and a single minimum.
• Topology trivialization is leveraged to design an annealing schedule that starts in the polynomial regime and transitions to the original landscape.
• The annealing schedule is implemented by gradually reducing noise magnitude during training, starting from a level that simplifies the landscape.
• Experiments are conducted on fully-connected and convolutional networks to evaluate training speed and generalization performance.

Experimental results

Research questions

• RQ1 How does additive gradient noise alter the topology of the loss landscape in deep networks?
• RQ2 What is the relationship between noise magnitude and the number of critical points in the energy landscape?
• RQ3 Can topology trivialization be harnessed to design effective noise annealing schedules?
• RQ4 Does noise annealing based on trivialization improve training speed and generalization?
• RQ5 How does the proposed method compare to standard training with fixed noise or no noise?

Key findings

• The energy landscape transitions from highly non-convex with exponentially many critical points to a regime with polynomially many critical points as noise magnitude increases.
• At high noise levels, the landscape trivializes to exactly one minimum, indicating a dramatic simplification of optimization dynamics.
• The proposed annealing scheme, which begins in the polynomial regime and transitions to the original landscape, accelerates training compared to standard methods.
• Generalization error is improved under the annealing scheme, indicating better generalization performance.
• The method is effective across both fully-connected and convolutional neural networks, demonstrating broad applicability.