[Paper Review] A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights

Weijie Su, Stephen Boyd+1 more|arXiv (Cornell University)|Mar 4, 2015

Sparse and Compressive Sensing TechniquesEngineering22 references544 citations

TL;DR

This paper derives a second-order ordinary differential equation (ODE) that models Nesterov's accelerated gradient method in the limit of small step sizes, revealing its dynamics as a damped oscillatory system. The ODE provides theoretical insight into the method's convergence behavior and enables the design of a restarting strategy that achieves linear convergence for strongly convex functions.

ABSTRACT

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus can serve as a tool for analysis. We show that the continuous time ODE allows for a better understanding of Nesterov's scheme. As a byproduct, we obtain a family of schemes with similar convergence rates. The ODE interpretation also suggests restarting Nesterov's scheme leading to an algorithm, which can be rigorously proven to converge at a linear rate whenever the objective is strongly convex.

Motivation & Objective

• To derive a continuous-time ODE that models Nesterov's accelerated gradient method in the small step size limit.
• To provide a theoretical framework for understanding the dynamics and convergence of Nesterov's scheme through ODE analysis.
• To show that the ODE captures key features such as momentum-induced oscillations and the role of the damping coefficient 3.
• To use the ODE to derive a restarting strategy that ensures linear convergence for strongly convex objectives.
• To generalize the method by identifying a family of schemes with similar convergence rates.

Proposed method

• Derive a second-order ODE, $\ddot{X} + \frac{3}{t}\dot{X} + \nabla f(X) = 0$, as the continuous limit of Nesterov's scheme under small step sizes.
• Establish approximate equivalence between the discrete Nesterov scheme and the continuous ODE through asymptotic analysis and numerical validation.
• Analyze the ODE as a damped harmonic oscillator, interpreting the $3/t$ damping term as governing the transition from overdamped to underdamped behavior.
• Use energy-based analysis and Lyapunov functions to prove convergence rates for both the ODE and its discrete counterpart.
• Propose a restarting strategy based on the ODE’s oscillatory behavior, where the algorithm is reset when the trajectory begins to overshoot.
• Demonstrate that the restarted scheme achieves linear convergence for strongly convex functions, with a rate dependent on the condition number.

Experimental results

Research questions

• RQ1 How can Nesterov’s accelerated gradient method be modeled as a continuous-time second-order ODE in the small step size limit?
• RQ2 What dynamical insights does the ODE provide regarding the oscillatory behavior and momentum effects in Nesterov’s scheme?
• RQ3 Why does the coefficient 3 appear in the damping term of the ODE, and what is its role in convergence?
• RQ4 Can the ODE framework be used to design a provably faster restarting strategy for Nesterov’s method?
• RQ5 What convergence rates can be achieved by schemes derived from the ODE, especially in the strongly convex case?

Key findings

• The continuous-time ODE $\ddot{X} + \frac{3}{t}\dot{X} + \nabla f(X) = 0$ is the exact limit of Nesterov’s accelerated gradient method as step sizes tend to zero.
• The ODE exhibits an inverse quadratic convergence rate: $f(X(t)) - f^* \leq O(\|x_0 - x^* olimits^2 / t^2)$, matching the discrete scheme’s rate.
• The damping coefficient 3 in the $3/t$ term explains the transition from overdamped to underdamped behavior, which corresponds to the oscillatory trajectories observed in practice.
• The ODE framework reveals that Nesterov’s scheme behaves like a system with momentum that initially suppresses oscillations but later leads to underdamping and overshoot.
• A restarting strategy based on the ODE’s dynamics achieves linear convergence for strongly convex functions, with a rate of $O(\|x_0 - x^* olimits^2 / (s^{3/2} \mu^{1/2} k^3))$, improving upon the non-restarted scheme.
• The ODE interpretation leads to a family of schemes with similar convergence rates, generalizing Nesterov’s method beyond the original coefficient choice.