[Paper Review] Activity Identification and Local Linear Convergence of Inertial Forward-Backward Splitting

Jingwei Liang, Jalal Fadili+1 more|arXiv (Cornell University)|Mar 12, 2015

Sparse and Compressive Sensing Techniques8 references3 citations

TL;DR

This paper proposes a unified analysis of inertial Forward-Backward (iFB) splitting algorithms for minimizing composite convex functions, where one component is Lipschitz-differentiable and the other is partly smooth relative to an active manifold 𝒩. It establishes finite active manifold identification and local linear convergence, providing theoretical justification for observed numerical behavior in problems like Lasso, group Lasso, and total variation minimization.

ABSTRACT

We consider the class of inertial Forward--Backward (iFB) proximal splitting algorithms, to minimize the sum of two proper lower semi-continuous convex functions, one of which having a Lipschitz continuous gradient and the other being partly smooth relative to an active manifold $\mathcal{M}$. Special cases of this class include the FB and, for an appropriate choice of the inertial parameter, FISTA-like schemes. We propose a unified analysis, under which we show that iFB-type splitting, (i) correctly identifies the active manifold $\mathcal{M}$ in a finite number of iterations, and then (ii) enters a local (linear) convergence regime, which is characterised precisely. This gives a grounded justification to the typical behaviour that has been observed numerically for many problems encompassed in our framework, including the Lasso, the group Lasso, total variation minimization and the nuclear norm regularization to name a few. These results may have numerous applications including in signal/image processing processing and machine learning.

Motivation & Objective

• To provide a unified theoretical framework for inertial Forward-Backward splitting in composite convex optimization.
• To explain the finite identification of the active manifold in iFB-type algorithms, a phenomenon commonly observed in practice.
• To establish local linear convergence of iFB schemes after active manifold identification, under partial smoothness and Lipschitz gradient assumptions.
• To justify the empirical success of FISTA-like methods and inertial schemes in problems such as Lasso, group Lasso, and nuclear norm regularization.
• To offer a precise characterization of the local convergence regime following active manifold identification.

Proposed method

• The analysis leverages the concept of partial smoothness relative to an active manifold 𝒩, which allows for a structured characterization of the problem's geometry.
• The method employs an inertial forward-backward splitting scheme with an adaptive inertial parameter, generalizing standard FISTA and FB methods.
• Finite active manifold identification is proven by analyzing the trajectory of iterates and showing that they eventually enter and remain within a neighborhood of the active manifold.
• Local linear convergence is established by exploiting the error bound property and the partial smoothness structure, leading to a linear rate of convergence after identification.
• The analysis relies on tools from variational analysis and monotone operator theory, particularly the properties of the proximal operator and the gradient mapping.
• The framework is general enough to include well-known problems such as Lasso, group Lasso, total variation, and low-rank matrix recovery as special cases.

Experimental results

Research questions

• RQ1 Can inertial Forward-Backward splitting algorithms identify the correct active manifold in finite iterations for partially smooth composite problems?
• RQ2 What conditions ensure local linear convergence of iFB schemes after active manifold identification?
• RQ3 How does the inertial parameter influence the convergence behavior and manifold identification in iFB methods?
• RQ4 To what extent does the proposed framework unify the analysis of FISTA-like and standard FB schemes?
• RQ5 Can the theoretical convergence properties explain the observed numerical performance in problems like Lasso and nuclear norm minimization?

Key findings

• The iFB algorithm identifies the active manifold 𝒩 in a finite number of iterations under the given assumptions.
• After active manifold identification, the iterates converge locally linearly, with a rate that depends on the curvature and condition number of the problem on the manifold.
• The convergence rate is characterized precisely through the error bound property and the partial smoothness structure.
• The framework unifies the analysis of FISTA and standard FB schemes, showing that both fall under the same convergence regime when the inertial parameter is appropriately chosen.
• The results provide theoretical grounding for the observed fast convergence in practice for problems such as Lasso, group Lasso, and total variation minimization.
• The analysis applies to a broad class of problems including sparse and low-rank optimization, confirming the robustness of iFB-type methods in signal and image processing and machine learning.