[Paper Review] Bregman forward-backward splitting for nonconvex composite optimization: superlinear convergence to nonisolated critical points

Masoud Ahookhosh, Andreas Themelis+1 more|arXiv (Cornell University)|May 28, 2019

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes Bella, a Bregman forward-backward splitting method for nonconvex composite optimization that achieves locally superlinear convergence to nonisolated critical points. By leveraging the Bregman forward-backward envelope (BFBE), a penalty function with favorable first- and second-order properties and a nonlinear error bound under a Lojasiewicz-type condition, the algorithm ensures global convergence and superlinear rates when directions are suitably selected.

ABSTRACT

We introduce Bella, a locally superlinearly convergent Bregman forward-backward splitting method for minimizing the sum of two nonconvex functions, one of which satisfying a relative smoothness condition and the other one possibly nonsmooth. A key tool of our methodology is the Bregman forward-backward envelope (BFBE), an exact and continuous penalty function with favorable first- and second-order properties, and enjoying a nonlinear error bound when the objective function satisfies a Lojasiewicz-type property. The proposed algorithm is of linesearch type over the BFBE along candidate update directions, and converges subsequentially to stationary points, globally under a KL condition, and owing to the given nonlinear error bound can attain superlinear convergence rates even when the limit point is a nonisolated minimum, provided the directions are suitably selected.

Motivation & Objective

• To address the challenge of achieving superlinear convergence in nonconvex composite optimization where minima may be nonisolated.
• To develop a linesearch-based algorithm that ensures global convergence to stationary points under the Kurdyka-Lojasiewicz (KL) condition.
• To establish a nonlinear error bound for the Bregman forward-backward envelope (BFBE), enabling superlinear convergence even at nonisolated minima.
• To extend the applicability of Bregman methods beyond convex settings by incorporating relative smoothness and nonsmooth components.

Proposed method

• The method employs the Bregman forward-backward envelope (BFBE), an exact and continuous penalty function that encapsulates the composite objective.
• The BFBE is constructed to possess favorable first- and second-order properties, enabling efficient descent via linesearch.
• A nonlinear error bound is derived for the BFBE under a Lojasiewicz-type condition, which is critical for superlinear convergence.
• The algorithm performs linesearch over the BFBE along candidate update directions to ensure sufficient decrease.
• Convergence is established via the Kurdyka-Lojasiewicz (KL) inequality, ensuring global convergence to stationary points.
• Superlinear convergence is achieved when the update directions are appropriately selected, even at nonisolated critical points.

Experimental results

Research questions

• RQ1 Can a Bregman forward-backward method achieve superlinear convergence in nonconvex optimization when the limit point is nonisolated?
• RQ2 What conditions on the objective function ensure a nonlinear error bound for the Bregman forward-backward envelope (BFBE)?
• RQ3 How can the Bregman forward-backward envelope be leveraged to ensure global convergence and fast local rates in nonconvex settings?
• RQ4 What role does the relative smoothness condition play in enabling convergence guarantees for nonconvex composite problems?
• RQ5 Under what conditions can linesearch over the BFBE yield superlinear convergence despite nonisolated minima?

Key findings

• The Bregman forward-backward envelope (BFBE) is an exact and continuous penalty function with favorable first- and second-order properties.
• The BFBE satisfies a nonlinear error bound when the objective function satisfies a Lojasiewicz-type condition.
• Global convergence to stationary points is guaranteed under the Kurdyka-Lojasiewicz (KL) condition.
• Superlinear convergence rates are achieved even at nonisolated critical points, provided the update directions are suitably selected.
• The algorithm is linesearch-based over the BFBE, ensuring sufficient decrease and robust convergence behavior.
• The method applies to nonconvex composite problems where one function is relatively smooth and the other may be nonsmooth.