[Paper Review] The value function approach to convergence analysis in composite optimization

Edouard Pauwels|arXiv (Cornell University)|Apr 6, 2016

Advanced Optimization Algorithms Research23 references20 citations

TL;DR

This paper presents a global convergence analysis for a backtracking variant of the composite Gauss-Newton method using the value function approach under tameness assumptions. It establishes convergence to a critical point under mild geometric conditions, leveraging the nonsmooth Kurdyka-Łojasiewicz (KL) inequality for definable functions, offering a flexible and verifiable framework applicable to a broad class of nonconvex, nonsmooth composite optimization problems.

ABSTRACT

This works aims at understanding further convergence properties of first order local search methods with complex geometries. We focus on the composite optimization model which unifies within a simple formalism many problems of this type. We provide a general convergence analysis of the composite Gauss-Newton method under tameness assumptions (an extension of semi-algebraicity). Tameness is a very general condition satisfied by virtually all problems solved in practice. The analysis is based on recent progresses in understanding convergence properties of sequential convex programming methods through the value function.

Motivation & Objective

• Address the lack of flexible, globally applicable convergence guarantees for first-order methods in composite nonconvex optimization with complex geometries.
• Overcome the reliance on strong local growth conditions around accumulation points, which are often hard to verify in practice.
• Provide a general convergence analysis for the composite Gauss-Newton method using the value function framework, extending prior work on sequential convex programming.
• Integrate a general backtracking line search into the analysis to handle functions with merely locally Lipschitz continuous gradients.
• Establish convergence under the tameness condition (definability in an o-minimal structure), which implies the KL inequality and excludes pathological behaviors like wild oscillations.

Proposed method

• Formulate the composite optimization problem as minimizing $ g(F(x)) $, where $ F: \mathbb{R}^n \to \mathbb{R}^m $ is $ \mathscr{C}^2 $, $ g: \mathbb{R}^m \to \mathbb{R} $ is convex and finite-valued, and $ D \subset \mathbb{R}^n $ is closed and convex.
• Apply a backtracking variant of the composite Gauss-Newton method where the stepsize parameter $ \mu_k $ is adaptively increased until a sufficient decrease condition is satisfied.
• Define the value function $ V_\mu(x) = \min_{y \in D} g(F(x) + \nabla F(x)(y - x)) + \frac{\mu}{2} \|y - x\|^2 $, which encapsulates the local quadratic approximation of the problem.
• Use the nonsmooth Kurdyka-Łojasiewicz (KL) inequality on the value function to control the decay of the objective and ensure convergence.
• Employ a Lyapunov-type function $ \varphi(V_\mu(x_k)) $, where $ \varphi $ is concave and differentiable, to derive a descent inequality that links the decrease in value function to the stepsize and subgradient norm.
• Prove that the sequence $ \{x_k\} $ is bounded and Cauchy in a neighborhood of a limit point, leveraging summability of $ \|x_{k+1} - x_k\| $, leading to convergence of iterates.

Experimental results

Research questions

• RQ1 Can a global convergence guarantee be established for the composite Gauss-Newton method without requiring strong local growth conditions?
• RQ2 How can the value function framework be extended to handle backtracking line search in composite nonconvex optimization?
• RQ3 To what extent does the tameness assumption (definability in an o-minimal structure) ensure the validity of the KL inequality for convergence analysis?
• RQ4 Can the convergence analysis be made flexible enough to include functions with only locally Lipschitz continuous gradients?
• RQ5 What conditions ensure that the iterates of the backtracking Gauss-Newton method converge to a critical point of the composite problem?

Key findings

• The proposed backtracking composite Gauss-Newton method converges globally to a critical point of the composite optimization problem under tameness assumptions.
• The convergence is established via the value function approach, which allows handling nonconvex, nonsmooth, and complexly structured problems without requiring strong local regularity conditions.
• The analysis leverages the nonsmooth Kurdyka-Łojasiewicz (KL) inequality, which holds for definable functions, ensuring that the objective value decreases sufficiently at each iteration.
• The sequence $ \{x_k\} $ is shown to be Cauchy in a neighborhood of the limit point, implying convergence of the iterates to a fixed point of the proximal map.
• The stepsize parameter $ \mu_k $ remains bounded for large $ k $, and the series $ \sum \|x_{k+1} - x_k\| $ converges, which is key to proving convergence of the iterates.
• The method is robust to locally Lipschitz continuous gradients and does not require strong convexity or sharpness assumptions, making it applicable to a wide range of practical problems.