[Paper Review] Line Search and Trust-Region Methods for Convex-Composite Optimization

James V. Burke, Abraham Engle|arXiv (Cornell University)|Jun 13, 2018

Advanced Optimization Algorithms Research20 references4 citations

TL;DR

This paper proposes and analyzes three descent methods—backtracking line search, weak Wolfe line search, and trust-region—for solving convex-composite optimization problems of the form $\min_x h(c(x)) + g(x)$, where $h$ is convex, $g$ is closed proper convex, and $c$ is $C^1$-smooth. The key contribution is proving global convergence to first-order stationary points under mild assumptions, with convergence established via descent in the model function $\Delta f(x;d)$ and safeguards on stepsize and trust-region radius.

ABSTRACT

We consider descent methods for solving non-finite valued nonsmooth convex-composite optimization problems that employ Gauss-Newton subproblems to determine the iteration update. Specifically, we establish the global convergence properties for descent methods that use a backtracking line search, a weak Wolfe line search, or a trust-region update. All of these approaches are designed to exploit the structure associated with convex-composite problems.

Motivation & Objective

• Address the need for efficient, globally convergent algorithms for convex-composite optimization problems arising in machine learning and dynamics.
• Develop and analyze descent methods that exploit the structure of $h(c(x)) + g(x)$, where $g$ induces sparsity or constraints.
• Establish global convergence for three methods: backtracking line search, weak Wolfe line search, and trust-region, using Gauss-Newton-type subproblems.
• Provide convergence guarantees for infinite-valued, nonsmooth composite problems, extending prior work on finite-valued or proximal methods.

Proposed method

• Use Gauss-Newton subproblems to compute search directions $d^k$ that minimize $\Delta f(x^k; d) = h(c(x^k) + \nabla c(x^k)d) + g(x^k + d) - f(x^k)$ subject to $\|d\| \leq \eta_k$.
• Implement a backtracking line search with Armijo-type conditions using the model function $\Delta f(x;d)$ as a surrogate for the directional derivative.
• Adapt the weak Wolfe line search to nonsmooth, infinite-valued functions by using $\Delta f(x;d)$ to estimate descent and curvature conditions.
• Apply a trust-region strategy where the step size is controlled by a radius $\delta_k$, with adaptive adjustment based on predicted vs. actual decrease.
• Introduce a safeguarding mechanism: if the iterate remains within a fixed neighborhood for too long, the objective function would diverge to $-\infty$, contradicting assumptions.
• Use the model function $\Delta f(x;d)$ to define a descent condition $\Delta f(x^k; d^k) < 0$ and ensure sufficient decrease via parameters $\beta_1, \beta_2, \kappa_1, \kappa_2$.

Experimental results

Research questions

• RQ1 Can backtracking and weak Wolfe line search methods be adapted to infinite-valued, nonsmooth convex-composite problems with structured components?
• RQ2 Does a trust-region method based on Gauss-Newton subproblems ensure global convergence for such problems?
• RQ3 Can the use of $\Delta f(x;d)$ as a model function replace directional derivatives in line search and trust-region frameworks for nonsmooth functions?
• RQ4 What conditions ensure that the objective function decreases sufficiently and the iterates converge to a stationary point?
• RQ5 How do the proposed methods differ in convergence behavior and assumptions from proximal or standard smooth optimization methods?

Key findings

• The backtracking, weak Wolfe, and trust-region methods all ensure global convergence to a first-order stationary point under mild assumptions.
• Global convergence is established by contradiction: if iterates remain bounded and do not converge, the objective function would decrease without bound, violating the working hypothesis.
• The model function $\Delta f(x;d)$ is used to define descent and curvature conditions, replacing directional derivatives in nonsmooth settings.
• The trust-region method ensures sufficient decrease by controlling the stepsize via $\delta_k$, with adaptive refinement when the predicted decrease is not achieved.
• The weak Wolfe line search is adapted to infinite-valued problems by using $\Delta f(x;d)$ to estimate both sufficient decrease and curvature, enabling application to nonsmooth $g$.
• Every cluster point of the iterates is a first-order stationary point, as $\Delta^C_1 f(\bar{x}) = 0$ at any limit point $\bar{x}$, under the assumptions of Theorem 6.1.