[Paper Review] Linear Coupling of Gradient and Mirror Descent: A Novel, Simple Interpretation of Nesterov's Accelerated Method

Zeyuan Allen-Zhu, Lorenzo Orecchia|arXiv (Cornell University)|Jul 6, 2014

Stochastic Gradient Optimization Techniques28 references7 citations

TL;DR

This paper introduces a novel interpretation of Nesterov's accelerated gradient method through linear coupling of gradient descent and mirror descent, revealing a cleaner, more intuitive mechanism than Nesterov's original proof. The approach unifies primal and dual progress, enabling broader applicability beyond Nesterov's original framework.

ABSTRACT

First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. We observe that the performances of gradient and mirror descent are complementary, so that faster algorithms can be designed by LINEARLY COUPLING the two. We show how to reconstruct Nesterov's accelerated gradient methods using linear coupling, which gives a cleaner interpretation than Nesterov's original proofs. We also discuss the power of linear coupling by extending it to many other settings that Nesterov's methods cannot apply to.

Motivation & Objective

• To provide a new, intuitive interpretation of Nesterov's accelerated gradient method by unifying gradient descent and mirror descent.
• To demonstrate that linear coupling of these two descent types yields faster convergence than either method alone.
• To extend the applicability of accelerated methods beyond the constraints of Nesterov's original framework.

Proposed method

• The method linearly combines gradient descent steps, which drive primal progress, with mirror descent steps, which drive dual progress.
• It uses a convex combination of the two descent directions to update the iterate, balancing primal and dual improvements.
• The coupling is formalized via a parameterized update rule that generalizes Nesterov's momentum scheme.
• The approach is derived from a primal-dual perspective, treating the two descent types as complementary components.
• The framework is shown to recover Nesterov's accelerated method with a clearer geometric and algorithmic interpretation.
• The method is extended to settings where Nesterov's original analysis does not apply, such as non-Euclidean and composite optimization.

Experimental results

Research questions

• RQ1 How can Nesterov's accelerated method be reinterpreted through the lens of linear coupling between gradient and mirror descent?
• RQ2 What is the role of primal and dual progress in accelerating first-order optimization?
• RQ3 Can the linear coupling framework be generalized to problems outside the scope of Nesterov's original method?
• RQ4 What are the convergence guarantees of the linearly coupled method compared to standard accelerated schemes?
• RQ5 How does the coupling mechanism improve interpretability and extend applicability in non-standard optimization settings?

Key findings

• The linear coupling framework provides a cleaner and more intuitive interpretation of Nesterov's accelerated method than the original proof.
• The method achieves the optimal convergence rate of O(1/k²) for smooth convex optimization, matching Nesterov's result.
• The approach naturally extends to non-Euclidean and composite settings where Nesterov's method does not directly apply.
• The framework reveals that acceleration arises from a balanced interplay between primal and dual progress.
• The linear coupling mechanism enables systematic design of new accelerated algorithms beyond the scope of classical Nesterov-style methods.
• The method demonstrates that mirror descent and gradient descent are not just alternatives but complementary tools when linearly combined.