[Paper Review] Entropy-SGD: Biasing Gradient Descent Into Wide Valleys

Pratik Chaudhari, Anna Choromanska+7 more|arXiv (Cornell University)|Nov 6, 2016

Model Reduction and Neural Networks60 references114 citations

TL;DR

Entropy-SGD introduces a local-entropy based objective that biases gradient descent toward wide, flat valleys, improving generalization and training speed on deep networks. It uses a two-loop approach with inner Langevin dynamics to estimate local entropy before outer weight updates.

ABSTRACT

This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenvalues. We leverage upon this observation to construct a local-entropy-based objective function that favors well-generalizable solutions lying in large flat regions of the energy landscape, while avoiding poorly-generalizable solutions located in the sharp valleys. Conceptually, our algorithm resembles two nested loops of SGD where we use Langevin dynamics in the inner loop to compute the gradient of the local entropy before each update of the weights. We show that the new objective has a smoother energy landscape and show improved generalization over SGD using uniform stability, under certain assumptions. Our experiments on convolutional and recurrent networks demonstrate that Entropy-SGD compares favorably to state-of-the-art techniques in terms of generalization error and training time.

Motivation & Objective

• Motivate training deep networks by leveraging the local geometry of the energy landscape.
• Propose a local-entropy based objective that favors wide valleys over sharp minima.
• Develop an optimization algorithm (Entropy-SGD) that estimates local entropy via Langevin dynamics.
• Analyze smoothing effects and generalization guarantees under certain assumptions.
• Demonstrate empirical performance on CNNs, RNNs, and text/vision benchmarks.

Proposed method

• Define local entropy F(x, gamma) as the log-partition function of a modified Gibbs distribution focusing on a neighborhood of x.
• Derive the gradient of local entropy via expectations over a Gibbs distribution and approximate it with stochastic gradient Langevin dynamics (SGLD).
• Implement Entropy-SGD as an outer SGD loop that updates weights using the gradient of -F(x, gamma) with an inner loop of L SGLD steps to estimate the local-entropy gradient.
• Introduce a scoping parameter gamma that controls the neighborhood size and propose an exponential schedule to progressively focus on wider valleys.
• Discuss practical implementation details including SGLD steps, averaging μ, momentum, and learning-rate choices.
• Provide a theoretical perspective tying smoother local-entropy objectives to improved stability and generalization bounds.

Experimental results

Research questions

• RQ1 Does the local-entropy based objective lead to smoother optimization landscapes compared to standard SGD?
• RQ2 Can Entropy-SGD achieve better generalization by biasing searches toward wide valleys?
• RQ3 How does the scoping parameter gamma influence optimization dynamics and generalization?
• RQ4 What empirical gains in generalization and training speed are achievable on CNNs, RNNs, and language models?

Key findings

• The Hessian eigenspectrum at local minima across architectures shows many near-zero directions and few large positive curvatures, suggesting wide valleys generalize better.
• Entropy-SGD yields comparable or better generalization to baselines while often training faster, including a 2x speed-up on RNNs.
• Using SGLD to estimate the gradient of local entropy leads to a smoother effective loss landscape and improved stability-based generalization bounds.
• The method scales to deep networks on MNIST, CIFAR-10, and PTB, with competitive test errors and favorable training dynamics.
• A scoping schedule for gamma allows exploration at coarse scales and refinement at finer scales, contributing to efficiency and generalization.