[Paper Review] A Lyapunov Analysis of Momentum Methods in Optimization

Ashia Wilson, Benjamin Recht+1 more|arXiv (Cornell University)|Nov 8, 2016

Stochastic Gradient Optimization Techniques4 references139 citations

TL;DR

The paper shows an equivalence between estimate sequences and Lyapunov functions for momentum methods, develops a unified Lyapunov-based analysis in continuous and discrete time, and derives new and existing accelerated algorithms through discretizations of Bregman Lagrangians.

ABSTRACT

Momentum methods play a significant role in optimization. Examples include Nesterov's accelerated gradient method and the conditional gradient algorithm. Several momentum methods are provably optimal under standard oracle models, and all use a technique called estimate sequences to analyze their convergence properties. The technique of estimate sequences has long been considered difficult to understand, leading many researchers to generate alternative, "more intuitive" methods and analyses. We show there is an equivalence between the technique of estimate sequences and a family of Lyapunov functions in both continuous and discrete time. This connection allows us to develop a simple and unified analysis of many existing momentum algorithms, introduce several new algorithms, and strengthen the connection between algorithms and continuous-time dynamical systems.

Motivation & Objective

• Motivate a unified Lyapunov-based framework for momentum methods in optimization.
• Show the equivalence between estimate sequences and Lyapunov functions in continuous and discrete time.
• Derive and analyze discrete-time algorithms via discretizations of continuous-time dynamics from Bregman Lagrangians.
• Strengthen connections between optimization algorithms and continuous-time dynamical systems.

Proposed method

• Define the Bregman Lagrangian and a second Bregman Lagrangian with ideal scaling conditions to obtain Euler–Lagrange equations describing continuous-time dynamics.
• Construct time-varying Lyapunov functions for these dynamics to certify convergence rates.
• Discretize the continuous-time dynamics using explicit and implicit Euler schemes and map to practical algorithms.
• Introduce and analyze accelerated methods via different discretization schemes and a Lyapunov function that decreases per iteration.
• Provide general convergence guarantees of the form E_t or E_k decreasing, yielding O(1/β_t) or O(1/A_k) rates under ideal scaling.
• Discuss how various known accelerated methods arise as special cases under specific choices of the map G or gradient updates.

Experimental results

Research questions

• RQ1 Can estimate sequences used in momentum methods be replaced by Lyapunov functions to analyze convergence in both continuous and discrete time?
• RQ2 What is the precise link between continuous-time Bregman Lagrangian dynamics and discrete-time optimization algorithms?
• RQ3 How can discretizations of Euler–Lagrange equations yield accelerated optimization algorithms with provable convergence rates?
• RQ4 Under what conditions do Lyapunov functions certify convergence rates for momentum and accelerated methods?
• RQ5 How do existing accelerated methods fit into a unified Lyapunov and discretization framework?

Key findings

• A Lyapunov framework is developed that unifies continuous-time and discrete-time analyses of momentum methods.
• Two families of Bregman Lagrangians yield Euler–Lagrange equations whose Lyapunov functions give O-convergence rates when ideal scaling holds.
• Discrete-time Lyapunov functions lead to general O(1/A_k) convergence guarantees for implicit and accelerated schemes.
• Accelerated methods such as gradient/mirror descent and universal high-order methods emerge as discretizations with appropriate G updates and smoothness assumptions.
• The analysis shows equivalence between estimate sequences and Lyapunov functions, enabling a simpler, dynamical-systems perspective.