[Paper Review] Local and Global Convergence of an Inertial Version of Forward-Backward Splitting

Patrick R. Johnstone, Pierre Moulin|arXiv (Cornell University)|Feb 8, 2015

Sparse and Compressive Sensing Techniques38 references4 citations

TL;DR

This paper analyzes an inertial forward-backward splitting (I-FBS) algorithm for convex composite optimization, proving global weak convergence and local linear convergence for sparse optimization problems. It identifies the optimal momentum parameters for local convergence and proposes a practical adaptive method (FISTA-AdOPT) that inherits FISTA's global $O(1/k^2)$ rate while achieving optimal local convergence, outperforming standard FISTA in later stages.

ABSTRACT

A problem of great interest in optimization is to minimize a sum of two closed, proper, and convex functions where one is smooth and the other has a computationally inexpensive proximal operator. In this paper we analyze a family of Inertial Forward-Backward Splitting (I-FBS) algorithms for solving this problem. We first apply a global Lyapunov analysis to I-FBS and prove weak convergence of the iterates to a minimizer in a real Hilbert space. We then show that the algorithms achieve local linear convergence for "sparse optimization", which is the important special case where the nonsmooth term is the $\ell_1$-norm. This result holds under either a restricted strong convexity or a strict complimentary condition and we do not require the objective to be strictly convex. For certain parameter choices we determine an upper bound on the number of iterations until the iterates are confined on a manifold containing the solution set and linear convergence holds. The local linear convergence result for sparse optimization holds for the Fast Iterative Shrinkage and Soft Thresholding Algorithm (FISTA) due to Beck and Teboulle which is a particular parameter choice for I-FBS. In spite of its optimal global objective function convergence rate, we show that FISTA is not optimal for sparse optimization with respect to the local convergence rate. We determine the locally optimal parameter choice for the I-FBS family. Finally we propose a method which inherits the excellent global rate of FISTA but also has excellent local rate.

Motivation & Objective

• To analyze the convergence properties of an inertial forward-backward splitting (I-FBS) algorithm for minimizing a sum of smooth and nonsmooth convex functions.
• To establish global weak convergence of I-FBS under general conditions on momentum and step-size parameters.
• To prove local linear convergence for sparse optimization (i.e., $ρ\|x\|_1$ regularization), without requiring strict convexity.
• To identify the locally optimal momentum parameter choice that maximizes the local convergence rate.
• To develop a practical algorithm that combines FISTA’s global $O(1/k^2)$ convergence with optimal local linear convergence.

Proposed method

• Uses a global Lyapunov analysis to prove weak convergence of I-FBS iterates to a minimizer in a real Hilbert space.
• Applies Lyapunov techniques to derive conditions under which I-FBS achieves local linear convergence for $ρ\|x\|_1$-regularized problems.
• Derives the locally optimal momentum parameter via the smallest eigenvalue of $A_E^T A_E$ on the active set $E$.
• Proposes FISTA-AdOPT: a practical algorithm that switches from Beck and Teboulle’s FISTA parameters to the locally optimal I-FBS parameters once objective function values begin to increase.
• Estimates the optimal momentum in practice using the support of the current iterate as a surrogate for the true active set $E$.
• Employs a restart mechanism when $F(x^k) > F(x^{k-1})$, signaling that the solution manifold has been reached.

Experimental results

Research questions

• RQ1 Does the inertial forward-backward splitting method achieve global weak convergence under general momentum and step-size conditions?
• RQ2 Can local linear convergence be established for sparse optimization problems without requiring the full objective to be strictly convex?
• RQ3 What is the optimal choice of momentum parameter that maximizes the local convergence rate in sparse optimization?
• RQ4 Can a practical algorithm be designed that inherits both FISTA’s global $O(1/k^2)$ convergence and the optimal local linear convergence rate?
• RQ5 How does the performance of FISTA-AdOPT compare to FISTA and adaptive restart variants in terms of iteration complexity and oscillation behavior?

Key findings

• The I-FBS algorithm achieves global weak convergence under the conditions $0 \leq \alpha_k \leq 1$ and $\limsup \alpha_k < 1$, with step-sizes satisfying $0 < \lambda_k \leq 1/L$.
• For sparse optimization, I-FBS achieves local linear convergence under either restricted strong convexity or strict complementarity, even without global strong convexity.
• The locally optimal momentum parameter is derived as $\alpha_{\text{opt}} = \frac{1 - \sqrt{l_E / L}}{1 + \sqrt{l_E / L}}$, where $l_E$ is the smallest eigenvalue of $A_E^T A_E$.
• FISTA is not optimal for sparse optimization in terms of local convergence rate, despite its optimal global $O(1/k^2)$ objective function convergence.
• FISTA-AdOPT achieves the optimal asymptotic iteration complexity, matching the theoretical lower bound derived in Corollary 8, and avoids oscillations seen in FISTA-AdRe.
• Numerical experiments show that FISTA-AdOPT and FISTA-AdRe both inherit FISTA’s global $O(1/k^2)$ rate and achieve optimal local convergence, with FISTA-AdOPT being more stable due to fixed momentum after adaptation.