[Paper Review] Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration

Soham De, Anirbit Mukherjee+1 more|arXiv (Cornell University)|Jul 18, 2018

Stochastic Gradient Optimization TechniquesComputer Science37 references82 citations

TL;DR

The paper provides convergence guarantees for RMSProp and ADAM in smooth non-convex optimization and empirically compares them to Nesterov acceleration on autoencoders and CIFAR-10. It also analyzes hyperparameter sensitivity, especially the momentum parameter in ADAM.

ABSTRACT

RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time. Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter $β_1$. We show that at very high values of the momentum parameter ($β_1 = 0.99$) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's $β_1$ is set to the most commonly used value: $β_1 = 0.9$, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Motivation & Objective

• Provide first convergence guarantees for adaptive gradient methods (RMSProp and ADAM) in non-convex optimization.
• Derive run-time bounds for reaching approximate criticality under smoothness assumptions.
• Empirically compare RMSProp and ADAM with Nesterov’s Accelerated Gradient on autoencoders and CIFAR-10.
• Highlight hyperparameter sensitivity, especially ADAM's momentum parameter β1, and generalization trends.

Proposed method

• Define L-smooth non-convex objectives and finite-sum structure f(x)=k^{-1} sum_p f_p(x).
• Introduce and analyze RMSProp and ADAM updates under deterministic and stochastic settings.
• Prove convergence to approximate critical points for stochastic RMSProp under a technical oracle assumption.
• Compare with Nesterov Accelerated Gradient (NAG) through experiments on autoencoders and VGG-9 on CIFAR-10.
• Utilize a diagonal pre-conditioner framework and corresponding convergence arguments for adaptive methods.

Experimental results

Research questions

• RQ1 Do RMSProp and ADAM converge to approximate critical points in smooth non-convex optimization?
• RQ2 What are the run-time bounds for these adaptive methods to reach near-stationarity?
• RQ3 How do RMSProp and ADAM compare to Nesterov acceleration in terms of training and generalization on neural nets?
• RQ4 How does the momentum parameter β1 affect ADAM’s performance and generalization?
• RQ5 Do adaptive methods generalize differently from non-adaptive methods as network size scales?

Key findings

• First convergence guarantees are established for adaptive gradient methods (RMSProp and ADAM) to reach approximate criticality in smooth non-convex objectives.
• Stochastic RMSProp convergence is shown under an additional assumption on the gradient oracle.
• Empirical results indicate high sensitivity of ADAM to the momentum parameter β1, with β1=0.99 often matching or outperforming carefully tuned NAG on some tasks.
• In full-batch and large-net regimes, ADAM with large β1 can give lower training and test losses compared to NAG and RMSProp on autoencoders.
• NAG tends to reduce gradient norms and produce iterates with increasing minimum Hessian eigenvalue trends on autoencoders.
• On CIFAR-10 with VGG-9, the empirical comparison is extended beyond autoencoders.