[Paper Review] RMSProp and equilibrated adaptive learning rates for non-convex optimization.

Yann Dauphin, Harm de Vries+2 more|arXiv (Cornell University)|Feb 15, 2015

Stochastic Gradient Optimization TechniquesComputer Science7 references183 citations

TL;DR

This paper proposes ESGD, an unbiased stochastic estimator of an equilibration preconditioner that improves adaptive learning rates for non-convex optimization by accounting for negative Hessian eigenvalues. Unlike RMSProp, which is a biased approximation of this preconditioner, ESGD provides more accurate step directions and outperforms both RMSProp and SGD in convergence speed while adding minimal computational overhead.

ABSTRACT

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes, i.e., diagonal preconditioners. We show that the optimal preconditioner is based on taking the absolute value of the Hessian's eigenvalues, which is not what Newton and classical preconditioners like Jacobi's do. In this paper, we propose a novel adaptive learning rate scheme based on the equilibration preconditioner and show that RMSProp approximates it, which may explain some of its success in the presence of saddle points. Whereas RMSProp is a biased estimator of the equilibration preconditioner, the proposed stochastic estimator, ESGD, is unbiased and only adds a small percentage to computing time. We find that both schemes yield very similar step directions but that ESGD sometimes surpasses RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Motivation & Objective

• To address the challenge of ill-conditioning in training deep neural networks using adaptive learning rates.
• To investigate how negative eigenvalues of the Hessian influence optimization dynamics in non-convex settings.
• To design a more accurate adaptive learning rate scheme by modeling the absolute values of Hessian eigenvalues, rather than relying on classical Newton or Jacobi preconditioners.
• To develop a stochastic estimator that is unbiased and computationally efficient for large-scale deep learning.

Proposed method

• Proposes the equilibration preconditioner, which uses the absolute values of Hessian eigenvalues to stabilize optimization in the presence of saddle points.
• Derives ESGD as an unbiased stochastic estimator of the equilibration preconditioner, improving upon RMSProp’s biased estimation.
• Uses a running average of squared gradients to approximate the inverse of the Hessian’s absolute eigenvalues, similar to RMSProp but with a correction for bias.
• Introduces a novel update rule that maintains the computational efficiency of RMSProp while ensuring unbiased estimation of the preconditioner.
• Employs a diagonal preconditioning strategy that adapts learning rates per parameter based on the equilibration principle.
• Analyzes the relationship between RMSProp and the equilibration preconditioner, showing RMSProp as a biased approximation of the ideal scheme.

Experimental results

Research questions

• RQ1 How do negative Hessian eigenvalues affect the performance of adaptive learning rate methods in non-convex optimization?
• RQ2 Can a more accurate preconditioner be derived by considering the absolute values of Hessian eigenvalues rather than their signs?
• RQ3 Why does RMSProp perform well in saddle point environments despite being a biased estimator of the ideal preconditioner?
• RQ4 Can an unbiased stochastic estimator of the equilibration preconditioner be designed with minimal computational cost?
• RQ5 Does the proposed method, ESGD, converge faster than RMSProp and SGD in practice?

Key findings

• ESGD is an unbiased estimator of the equilibration preconditioner, unlike RMSProp, which is a biased approximation of the same ideal scheme.
• Both ESGD and RMSProp produce very similar step directions during optimization, indicating that RMSProp's success is partly due to approximating the equilibration principle.
• ESGD consistently converges faster than plain stochastic gradient descent across tested scenarios.
• ESGD sometimes surpasses RMSProp in convergence speed, demonstrating that unbiased estimation leads to improved optimization dynamics.
• The computational overhead of ESGD is minimal, adding only a small percentage to training time compared to RMSProp.
• The equilibration preconditioner, based on absolute Hessian eigenvalues, is theoretically superior to Newton and Jacobi methods in non-convex settings with saddle points.