[Paper Review] Anderson Acceleration of Proximal Gradient Methods

Vien Van, Mikael Johansson|arXiv (Cornell University)|Oct 18, 2019

Numerical methods in inverse problems18 citations

TL;DR

This paper introduces a stabilized Anderson acceleration framework for proximal gradient methods, enabling local superlinear convergence while preserving global worst-case convergence guarantees. It establishes local convergence for non-smooth mappings, proves global convergence is unguaranteed for native Anderson acceleration, and applies the method to non-Euclidean Bregman proximal gradient schemes with empirical speedups of several orders of magnitude on real-world problems.

ABSTRACT

Anderson acceleration is a well-established and simple technique for speeding up fixed-point computations with countless applications. Previous studies of Anderson acceleration in optimization have only been able to provide convergence guarantees for unconstrained and smooth problems. This work introduces novel methods for adapting Anderson acceleration to (non-smooth and constrained) proximal gradient algorithms. Under some technical conditions, we extend the existing local convergence results of Anderson acceleration for smooth fixed-point mappings to the proposed scheme. We also prove analytically that it is not, in general, possible to guarantee global convergence of native Anderson acceleration. We therefore propose a simple scheme for stabilization that combines the global worst-case guarantees of proximal gradient methods with the local adaptation and practical speed-up of Anderson acceleration.

Motivation & Objective

• Address the lack of theoretical convergence guarantees for native Anderson acceleration in non-smooth, composite optimization problems.
• Stabilize Anderson acceleration to retain the global worst-case convergence of proximal gradient methods while enabling local acceleration.
• Extend Anderson acceleration to the Bregman proximal gradient framework, enabling applications in non-Euclidean geometry.
• Demonstrate significant practical speedups on constrained optimization problems, including logistic regression, nonnegative least squares, and relative-entropy regression.
• Provide theoretical justification for the limitations of native Anderson acceleration and propose a guard-step mechanism to ensure global convergence.

Proposed method

• Propose a stabilized Anderson acceleration scheme for the proximal gradient algorithm (PGA) that combines AA steps with a safeguarded backtracking line search to maintain global convergence.
• Introduce a guard step that reverts to standard PGA when AA steps fail to reduce the objective, ensuring global worst-case convergence is preserved.
• Adapt Anderson acceleration to the Bregman proximal gradient (BPG) family using a Bregman divergence as the proximity measure, enabling application to non-Euclidean geometries.
• Use a minimal polynomial extrapolation approach to estimate the fixed-point of the proximal operator, leveraging past iterates to accelerate convergence.
• Formulate the AA update as a convex subproblem that minimizes the residual in a Krylov subspace, ensuring numerical stability and efficiency.
• Apply the method to various non-smooth, constrained optimization problems, including L1-regularized logistic regression and nonnegative matrix factorization subproblems.

Experimental results

Research questions

• RQ1 Can Anderson acceleration be theoretically justified for non-smooth fixed-point mappings arising in proximal gradient methods?
• RQ2 Is global convergence of native Anderson acceleration guaranteed for non-smooth or composite optimization problems?
• RQ3 How can Anderson acceleration be stabilized to preserve the global worst-case convergence of proximal gradient methods while enabling local acceleration?
• RQ4 Can Anderson acceleration be extended to non-Euclidean optimization frameworks such as Bregman proximal gradient methods?
• RQ5 What empirical performance gains can be achieved by combining Anderson acceleration with proximal gradient methods on real-world constrained optimization problems?

Key findings

• The authors prove that native Anderson acceleration cannot guarantee global convergence for non-smooth mappings, providing a counterexample with a strongly convex problem where AA fails to converge.
• The proposed stabilized AA-PGA method achieves global convergence with the same worst-case iteration complexity as standard PGA, while enabling local superlinear convergence.
• On constrained logistic regression problems, the stabilized AA-PGA method achieves speedups of up to several orders of magnitude compared to standard PGA and other first-order methods.
• For nonnegative least squares problems, AA-BPG achieves finite-time convergence in some hard instances, suggesting it can identify solutions within the span of past iterates.
• In relative-entropy nonnegative regression, AA-BPG achieves linear convergence even in sublinearly converging hard instances, demonstrating strong local adaptation to problem structure.
• The method is the first to successfully apply Anderson acceleration to non-Euclidean geometry via the Bregman proximal gradient framework, with promising empirical results across diverse problem classes.