[Paper Review] Accelerated Gradient Methods with Memory

Ross Drummond, Stephen Duncan|arXiv (Cornell University)|May 23, 2018

Advanced Optimization Algorithms Research14 references4 citations

TL;DR

This paper proposes a family of accelerated first-order optimization algorithms, denoted $\Sigma_N$, that leverage memory of past iterates to achieve faster convergence for strongly convex functions. By extending the fast gradient method with $N$-step history and incorporating an adaptive restarting controller, the method maintains acceleration while restoring robustness, achieving a function value of $7.58 \times 10^{-12}$ on the Rosenbrock function in 43 iterations from $(-1,1)$.

ABSTRACT

A set of accelerated first order algorithms with memory are proposed for minimising strongly convex functions. The algorithms are differentiated by their use of the iterate history for the gradient step. The increased convergence rate of the proposed algorithms comes at the cost of robustness, a problem that is resolved by a switching controller based upon adaptive restarting. Several numerical examples highlight the benefits of the proposed approach over the fast gradient method. For example, it is shown that these gradient based methods can minimise the Rosenbrock banana function to $7.58 imes 10^{-12}$ in 43 iterations from an initial condition of $(-1,1)$.

Motivation & Objective

• To develop first-order optimization algorithms that accelerate convergence beyond the fast gradient method by utilizing historical iterate information.
• To address the inherent lack of robustness in accelerated methods when applied to ill-conditioned or non-ideal problems.
• To generalize the convergence analysis of accelerated methods from quadratic to strongly convex functions using absolute stability theory.
• To design a switching-based adaptive restarting controller that restores monotonic convergence while preserving acceleration.
• To demonstrate the effectiveness of the proposed algorithms on both convex and non-convex benchmark functions, including the Rosenbrock and Rastrigin functions.

Proposed method

• The proposed algorithm $\Sigma_N$ generalizes the fast gradient method by incorporating $N$-step history of iterates to form a momentum-like update, enhancing convergence speed.
• The algorithm is parameterized for quadratic functions such that the error in the dominant eigenspace decays at rate $\left(1 - \left(\frac{\mu}{L}\right)^{1/N}\right)^k$.
• A switching controller based on adaptive restarting is introduced to detect instability and switch to a stable iterate, restoring monotonic convergence.
• The stability and convergence properties are analyzed using absolute stability theory, extending results from quadratic to strongly convex problems.
• The controller dynamically selects among candidate iterates at each step, ensuring robustness without sacrificing acceleration.
• The method is applied to non-convex functions like the Rastrigin function, where it successfully escapes local minima and converges near the global minimum.

Experimental results

Research questions

• RQ1 Can incorporating $N$-step history into first-order methods significantly accelerate convergence for strongly convex functions?
• RQ2 Why do accelerated methods like the fast gradient method lose robustness, and how can this be systematically addressed?
• RQ3 Can a switching-based adaptive restarting mechanism restore monotonic convergence while preserving the fast convergence rate?
• RQ4 How do memory-based algorithms perform on non-convex functions with multiple local minima, such as the Rastrigin function?
• RQ5 To what extent can the proposed method outperform classical methods like gradient descent and Nelder-Mead on challenging test functions like the Rosenbrock banana function?

Key findings

• The $\Sigma_N$ algorithm achieves a convergence rate of $\left(1 - \left(\frac{\mu}{L}\right)^{1/N}\right)^k$ for the dominant error mode, which is faster than the fast gradient method when $N \geq 3$.
• On the Rosenbrock banana function, $\Sigma_9^{ml}$ achieved a function value of $7.58 \times 10^{-12}$ in 43 iterations from $x_0 = (-1,1)$, outperforming the Nelder-Mead method which required 185 iterations to reach $1.35 \times 10^{-10}$.
• The fast gradient method was found to be divergent for the same Rosenbrock instance due to poor robustness, while $\Sigma_N^{ml}$ remained stable and convergent.
• For the Rastrigin function, $\Sigma_6^{ml}$ achieved function values between $10^{-6}$ and $10^{-8}$ from favorable initial conditions along a star-shaped pattern, while other regions yielded only $10^{-3}$ to $10^{-1}$.
• From $x_0 = (5,5)$, the algorithm reached $f_{\text{Rast}} = 10^{-6}$ in 463 iterations, demonstrating ability to escape local minima and converge near the global minimum.
• The adaptive restarting controller successfully restored monotonic convergence, enabling the algorithm to maintain acceleration without instability, even when $L$ exceeded the local curvature at the minimizer.