[Paper Review] Equilibrated adaptive learning rates for non-convex optimization

Yann Dauphin, Harm de Vries+1 more|arXiv (Cornell University)|Feb 15, 2015

Stochastic Gradient Optimization TechniquesComputer Science18 references152 citations

TL;DR

This paper proposes ESGD, an adaptive learning rate method for non-convex optimization that uses the equilibration preconditioner to better handle saddle points in deep learning. By aligning update directions more closely with the Hessian's curvature structure than RMSProp or Jacobi preconditioning, ESGD achieves faster convergence—3× faster than SGD on MNIST—and outperforms RMSProp in training speed and final error, with empirical evidence showing RMSProp's success may stem from its similarity to equilibration.

ABSTRACT

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so-called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Motivation & Objective

• To address the challenge of ill-conditioning in non-convex optimization, especially around saddle points, which slow down training in deep neural networks.
• To analyze the limitations of existing diagonal preconditioners—particularly the Jacobi preconditioner—when curvature includes both positive and negative eigenvalues.
• To propose a new adaptive learning rate schedule, ESGD, based on the equilibration preconditioner, which better handles mixed curvature in non-convex landscapes.
• To empirically validate that ESGD outperforms RMSProp and SGD in convergence speed and final loss on deep autoencoder benchmarks.
• To investigate why RMSProp performs well in practice by comparing its update direction to that of equilibrated updates.

Proposed method

• The equilibration preconditioner is defined as $ \mathbf{D}^{\text{E}} = \sqrt{\text{diag}(\mathbf{H}^2)} $, where $ \mathbf{H} $ is the Hessian, to balance curvature across both positive and negative directions.
• ESGD applies this preconditioner adaptively by estimating $ \mathbf{D}^{\text{E}} $ using unbiased stochastic estimators from mini-batches of gradients.
• The method transforms the parameter space via $ \hat{\theta} = \mathbf{D}^{1/2} \theta $, leading to a new function $ \hat{f} $ with more uniform curvature, improving convergence.
• Gradient descent is applied in the transformed space using the preconditioned update: $ \theta_t = \theta_{t-1} - \eta \mathbf{D}^{-1} \nabla f(\theta) $.
• The approach avoids full Hessian computation by using diagonal approximations, making it computationally efficient for large-scale models.
• The method is evaluated on deep autoencoders using Theano, with training error reported without momentum to isolate optimizer performance.

Experimental results

Research questions

• RQ1 How does the Jacobi preconditioner perform in the presence of both positive and negative curvature typical of saddle points in non-convex optimization?
• RQ2 Why does RMSProp achieve strong performance in training deep networks, despite not being explicitly designed for saddle point escape?
• RQ3 Can the equilibration preconditioner provide better convergence behavior than Jacobi or RMSProp in non-convex settings?
• RQ4 To what extent do the update directions of RMSProp align with those of the equilibration preconditioner during training?
• RQ5 Does an adaptive learning rate method based on equilibration outperform RMSProp and SGD in terms of convergence speed and final loss?

Key findings

• ESGD significantly outperforms RMSProp and SGD on the MNIST autoencoder benchmark, achieving a final training MSE of 0.86 compared to SGD’s 2.1 and RMSProp’s higher values.
• On MNIST, ESGD converges approximately three times faster than standard SGD, with performance gains becoming most apparent after 250 epochs.
• On the CURVES dataset, ESGD performs best, though the performance gap with RMSProp is smaller, suggesting dataset-dependent behavior of preconditioners.
• The cosine distance between RMSProp’s estimated preconditioner and the equilibration matrix remains low (~0.05) on CURVES, indicating strong alignment in update direction.
• On MNIST, RMSProp diverges from equilibration after 1000 epochs, coinciding with a drop in performance relative to ESGD, suggesting that RMSProp’s success is linked to its similarity to equilibration.
• Theoretical and empirical analysis confirms that the Jacobi preconditioner performs poorly in mixed-curvature settings, while equilibration better balances positive and negative curvature for faster escape from saddle points.