[Paper Review] A Robust Accelerated Optimization Algorithm for Strongly Convex Functions

Saman Cyrus, Bin Hu+2 more|arXiv (Cornell University)|Oct 12, 2017

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper introduces the Robust Momentum Method, a first-order optimization algorithm for smooth strongly convex functions that balances convergence speed and robustness to gradient noise via a single tunable parameter. By leveraging control-theoretic design, the method achieves faster convergence than the Gradient Method while maintaining stability under noisy gradients, with performance adjustable from the fastest (Triple Momentum Method) to the most robust (Gradient Method).

ABSTRACT

This work proposes an accelerated first-order algorithm we call the Robust Momentum Method for optimizing smooth strongly convex functions. The algorithm has a single scalar parameter that can be tuned to trade off robustness to gradient noise versus worst-case convergence rate. At one extreme, the algorithm is faster than Nesterov's Fast Gradient Method by a constant factor but more fragile to noise. At the other extreme, the algorithm reduces to the Gradient Method and is very robust to noise. The algorithm design technique is inspired by methods from classical control theory and the resulting algorithm has a simple analytical form. Algorithm performance is verified on a series of numerical simulations in both noise-free and relative gradient noise cases.

Motivation & Objective

• Address the trade-off between convergence speed and robustness to gradient noise in first-order optimization for strongly convex functions.
• Develop an accelerated algorithm that maintains fast convergence while being resilient to inexact gradient evaluations and uncertain problem parameters.
• Provide a unified framework that interpolates between the Fast Gradient Method and the Gradient Method through a single parameter, enabling systematic tuning.
• Ensure theoretical convergence guarantees under both noise-free and noisy gradient conditions using Lyapunov analysis and linear matrix inequalities.
• Demonstrate that the algorithm's robustness margin increases with slower convergence rates, enabling stability under high noise levels.

Proposed method

• Propose a three-term recurrence: $ x_{k+1} = x_k + \beta(x_k - x_{k-1}) - \alpha \nabla f(y_k) $, with $ y_k = x_k + \gamma(x_k - x_{k-1}) $, parameterized by $ \rho $, the worst-case convergence rate.
• Derive explicit formulas for $ \alpha, \beta, \gamma $ in terms of $ \rho, L, m $, and condition number $ \kappa = L/m $, ensuring linear convergence with rate $ \rho $.
• Use a Lyapunov function $ V_k $ to prove convergence, where $ V_{k+1} \leq \rho^2 V_k - \nu \|\nabla g(y_k)\|^2 $, with $ \nu $ increasing as $ \rho $ increases.
• Tune the parameter $ \nu \in [0, 1 - \frac{1}{2\kappa}] $ to control the robustness margin, where higher $ \nu $ increases tolerance to gradient noise.
• Apply a numerical rate analysis via linear matrix inequalities to compute worst-case convergence rates under relative gradient noise $ \|r_k\| \leq \delta \|\nabla f(y_k)\| $.
• Validate performance through simulations on a 2D quadratic function with varying noise levels, comparing to FGM and GM.

Experimental results

Research questions

• RQ1 How can first-order optimization methods be designed to balance fast convergence and robustness to gradient noise in strongly convex problems?
• RQ2 Can a single-parameter algorithm interpolate between the fastest known convergence rate and the most robust behavior in first-order optimization?
• RQ3 What is the relationship between the convergence rate and the robustness margin under gradient noise, and can it be analytically quantified?
• RQ4 How does the proposed method compare to the Fast Gradient Method and Gradient Method in terms of stability under increasing noise levels?
• RQ5 Can theoretical convergence guarantees be maintained under relative gradient noise using a Lyapunov-based analysis with a tunable robustness parameter?

Key findings

• The Robust Momentum Method achieves a worst-case linear convergence rate of $ \rho = 1 - \frac{1}{\sqrt{\kappa}} $ when $ \nu = 0 $, matching the Triple Momentum Method and outperforming the Fast Gradient Method in noise-free conditions.
• When $ \nu = 1 - \frac{1}{2\kappa} $, the algorithm reduces to the Gradient Method with stepsize $ \alpha = \frac{1}{L} $, ensuring maximum robustness to gradient noise.
• For $ \delta = 0.25 $, the Robust Momentum Method with $ \nu = 0.55 $ remains stable, while the Fast Gradient Method diverges, demonstrating superior robustness under moderate noise.
• The Fast Gradient Method has a faster convergence bound than the Robust Momentum Method for $ 0.26 < \delta < 0.41 $, but becomes unstable for $ \delta > 0.5 $, whereas the Robust Momentum Method remains stable up to $ \delta \to 1 $.
• The robustness margin $ \nu $ increases with $ \rho $, meaning slower convergence rates provide greater resilience to gradient noise, quantitatively linking performance and robustness.
• Simulations confirm that the Robust Momentum Method with $ \nu = 0 $ converges fastest in the noiseless case, but diverges under $ \delta > 0.13 $, while higher $ \nu $ values maintain stability even at $ \delta = 0.5 $.