[Paper Review] Acceleration via Symplectic Discretization of High-Resolution Differential Equations

Bin Shi, Simon S. Du+2 more|arXiv (Cornell University)|Feb 11, 2019

Stochastic Gradient Optimization Techniques15 references44 citations

TL;DR

The paper studies first-order optimization by discretizing high-resolution ODEs for Nesterov’s accelerated methods and Polyak’s heavy-ball method, showing symplectic discretization yields accelerated rates for strongly convex and convex objectives, unlike explicit or low-resolution discretizations.

ABSTRACT

We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme. We show that the optimization algorithm generated by applying the symplectic scheme to a high-resolution ODE proposed by Shi et al. [2018] achieves an accelerated rate for minimizing smooth strongly convex functions. On the other hand, the resulting algorithm either fails to achieve acceleration or is impractical when the scheme is implicit, the ODE is low-resolution, or the scheme is explicit.

Motivation & Objective

• Motivate and analyze whether discretizing high-resolution ODEs can yield accelerated first-order optimization methods.
• Compare three simple discretization schemes (symplectic Euler, explicit Euler, implicit Euler) on high- and low-resolution ODEs.
• Establish when acceleration is preserved or lost under discretization, using Lyapunov-based analysis.
• Clarify the role of high-resolution ODEs and gradient-correction terms in enabling stable, accelerated discretizations.

Proposed method

• Model acceleration methods by discretizing high-resolution ODEs for NAG-C, NAG-SC, and heavy-ball using three schemes: symplectic Euler (S), explicit Euler (E), and implicit Euler (I).
• Use phase-space formulations of the ODEs and derive discrete update rules for each scheme (S), (E), (I).
• Apply Lyapunov function analysis to obtain convergence rates for the discrete schemes.
• Compare acceleration behavior across high-resolution ODEs (NAG-SC, NAG-C) and a low-resolution ODE (heavy-ball).
• Demonstrate that symplectic discretization preserves acceleration for high-resolution ODEs, while explicit/implicit schemes show limitations.

Experimental results

Research questions

• RQ1 Does the symplectic Euler discretization preserve the acceleration properties when discretizing the high-resolution ODEs for NAG-SC and NAG-C?
• RQ2 Do explicit or implicit Euler discretizations achieve acceleration, and under what conditions are they practical?
• RQ3 How do discretizations of high-resolution ODEs compare to discretizations of low-resolution ODEs (e.g., heavy-ball) in terms of achieving acceleration?
• RQ4 What role do Lyapunov functions play in proving discrete-time acceleration results for these discretizations?
• RQ5 What implications do high-resolution ODEs and symplectic schemes have for designing new accelerated optimization algorithms?

Key findings

• The symplectic Euler discretization of the high-resolution NAG-SC ODE achieves acceleration with rate bounds like f(x_k)−f(x*) ≤ O(1)/(1+O(1)√(μ/L))^k for appropriate step sizes.
• Explicit Euler discretization fails to achieve acceleration for the high-resolution NAG-SC ODE, though it is simple to implement.
• Implicit Euler discretization attains acceleration but is generally impractical except in special cases (e.g., quadratic objectives).
• Discretizing the low-resolution heavy-ball ODE with any Euler scheme does not yield acceleration, highlighting the importance of high-resolution ODEs for acceleration.
• For convex functions, symplectic discretization similarly exhibits superior rates compared to the other two schemes, underscoring the role of gradient correction in high-resolution settings.
• The paper emphasizes that high-resolution ODEs and symplectic discretization enable stable, large-step, accelerated discrete-time methods.