[Paper Review] Understanding the Acceleration Phenomenon via High-Resolution Differential Equations

Bin Shi, Simon S. Du+2 more|arXiv (Cornell University)|Oct 21, 2018

Sparse and Compressive Sensing Techniques30 references86 citations

TL;DR

The paper introduces high-resolution ODEs to differentiate Nesterov’s accelerated gradient methods from the heavy-ball method, reveals a gradient correction mechanism, and shows a cubic rate for minimizing the squared gradient norm with NAG-C.

ABSTRACT

Gradient-based optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms---Nesterov's accelerated gradient method for strongly convex functions (NAG-SC) and Polyak's heavy-ball method---we study an alternative limiting process that yields high-resolution ODEs. We show that these ODEs permit a general Lyapunov function framework for the analysis of convergence in both continuous and discrete time. We also show that these ODEs are more accurate surrogates for the underlying algorithms; in particular, they not only distinguish between NAG-SC and Polyak's heavy-ball method, but they allow the identification of a term that we refer to as "gradient correction" that is present in NAG-SC but not in the heavy-ball method and is responsible for the qualitative difference in convergence of the two methods. We also use the high-resolution ODE framework to study Nesterov's accelerated gradient method for (non-strongly) convex functions, uncovering a hitherto unknown result---that NAG-C minimizes the squared gradient norm at an inverse cubic rate. Finally, by modifying the high-resolution ODE of NAG-C, we obtain a family of new optimization methods that are shown to maintain the accelerated convergence rates of NAG-C for smooth convex functions.

Motivation & Objective

• Motivate the study of gradient-based optimization through limiting ODEs and address the indistinguishability of NAG-SC and heavy-ball in traditional ODEs.
• Introduce a high-resolution ODE framework that retains O(sqrt{s}) terms to differentiate algorithms.
• Develop Lyapunov function techniques to analyze convergence in continuous and discrete time.
• Uncover the gradient correction term as essential for acceleration.
• Extend the analysis to NAG-C and derive new insights into gradient norm minimization rates.

Proposed method

• Derive high-resolution ODEs by including O(sqrt{s}) terms in the limiting process for NAG-SC, revealing the Hessian-driven gradient correction.
• Construct continuous-time Lyapunov functions for the high-resolution ODEs and prove their decrease along trajectories.
• Translate continuous Lyapunov analyses into discrete-time Lyapunov functions via a phase-space representation.
• Use the high-resolution ODE framework to compare NAG-SC and the heavy-ball method and explain acceleration differences.
• Show that the gradient correction term appears in discrete forms and affects convergence behavior.
• Apply the framework to NAG-C to establish a cubic rate for minimizing the squared gradient norm and discuss extensions.

Experimental results

Research questions

• RQ1 How do high-resolution ODEs distinguish NAG-SC from Polyak’s heavy-ball method?
• RQ2 What role does the Hessian-driven gradient correction play in acceleration for NAG-SC compared to the heavy-ball method?
• RQ3 Can a continuous-time Lyapunov analysis explain the discrete acceleration properties of NAG-SC?
• RQ4 What convergence rates can be obtained for NAG-C in terms of gradient norms, not only function values?
• RQ5 Can the high-resolution ODE framework yield new optimization methods that preserve acceleration?

Key findings

• High-resolution ODEs differentiate NAG-SC from the heavy-ball method by preserving the gradient correction term.
• The gradient correction term leads to adaptive damping via the Hessian, contributing to acceleration.
• A discrete Lyapunov function recovers the optimal linear convergence rate for NAG-SC.
• For NAG-C, the framework shows a new inverse cubic rate for minimizing the squared gradient norm.
• The high-resolution ODEs offer a more accurate surrogate for discretized algorithms than low-resolution ODEs.
• Modifications of the high-resolution ODE for NAG-C yield a family of methods maintaining accelerated rates for smooth convex functions.