[Paper Review] Variants of RMSProp and Adagrad with Logarithmic Regret Bounds

Mahesh Chandra Mukkamala, Matthias Hein|arXiv (Cornell University)|Jun 17, 2017

Advanced Bandit Algorithms ResearchDecision Sciences17 references110 citations

TL;DR

The paper analyzes RMSProp and Adagrad in online convex optimization, proving sqrt(T) regret for RMSProp and logarithmic regret for SC-Adagrad and SC-RMSProp, with experiments showing competitive performance in strongly convex and deep learning settings.

ABSTRACT

Adaptive gradient methods have become recently very popular, in particular as they have been shown to be useful in the training of deep neural networks. In this paper we have analyzed RMSProp, originally proposed for the training of deep neural networks, in the context of online convex optimization and show $\sqrt{T}$-type regret bounds. Moreover, we propose two variants SC-Adagrad and SC-RMSProp for which we show logarithmic regret bounds for strongly convex functions. Finally, we demonstrate in the experiments that these new variants outperform other adaptive gradient techniques or stochastic gradient descent in the optimization of strongly convex functions as well as in training of deep neural networks.

Motivation & Objective

• Motivate adaptive gradient methods for online convex optimization and deep learning.
• Establish theoretical regret bounds for RMSProp in general convex settings.
• Propose SC-Adagrad and SC-RMSProp for strongly convex functions with logarithmic regret.
• Demonstrate empirical performance improvements over standard adaptive methods and SGD on convex and deep learning tasks.

Proposed method

• Model RMSProp as a diagonal preconditioned update with weighted projections onto a convex set.
• Show Adagrad corresponds to a special case of RMSProp under certain parameter choices.
• Introduce SC-Adagrad with a time- and coordinate-wise damping delta_t to achieve O(log T) regret for strongly convex functions.
• Introduce SC-RMSProp as a strongly convex variant with analogous logarithmic regret guarantees.
• Derive data-dependent regret bounds and provide decay schemes for delta_t that ensure adaptivity and stability.

Experimental results

Research questions

• RQ1 Can RMSProp achieve regret bounds in online convex optimization, and under what weighting/biasing schemes?
• RQ2 Do variants SC-Adagrad and SC-RMSProp achieve logarithmic regret for strongly convex objectives?
• RQ3 How should damping and step-size schedules be chosen to balance adaptivity and stability in online settings?
• RQ4 Do the proposed methods outperform existing adaptive gradient methods and SGD on strongly convex problems and in neural network training?

Key findings

• RMSProp can be analyzed within online convex optimization and achieves a data-dependent O(sqrt(T)) regret bound under appropriate weighting.
• SC-Adagrad achieves a data-dependent logarithmic regret bound for strongly convex functions with a non-increasing damping delta_t.
• SC-RMSProp extends the SC-Adagrad analysis to RMSProp-like updates, yielding logarithmic regret for strongly convex objectives.
• A damping strategy delta_t (e.g., xi2 e^{-xi1 v_t,i}) and coordinate-wise adaptation improve stability and performance in practice.
• Experiments on three datasets show the new variants are competitive or outperform other adaptive methods and SGD for strongly convex optimization and neural network training.