[Paper Review] Cutting plane oracles for non-smooth trust-regions

Dominikus Noll|arXiv (Cornell University)|May 16, 2019

Advanced Optimization Algorithms Research38 references4 citations

TL;DR

This paper introduces a unified convergence framework for non-smooth trust-region methods using cutting plane oracles that satisfy four basic rules. It proves global convergence to critical points for a broad class of methods, including the method of downshifted tangents and splitting techniques, by establishing strict model conditions and extending convergence theory beyond previous assumptions, such as the need for positive definite second-order terms.

ABSTRACT

We prove global convergence of a bundle trust region algorithm for non-smooth non-convex optimization, where cutting planes are generated by oracles respecting four basic rules. The benefit is that convergence theory applies to a large variety of methods encountered in practice. This includes in particular the method of downshifted tangents, for which previously no convergence result in the trust region framework was known. We also show that certain splitting techniques can be seen as special cases of bundle trust region techniques.

Motivation & Objective

• To establish global convergence for non-smooth trust-region algorithms using a general class of cutting plane oracles.
• To resolve the open question of whether aggregation techniques from convex bundle methods can be applied in non-convex trust-region settings.
• To show that the method of downshifted tangents, previously without convergence proof in trust regions, is covered by the proposed framework.
• To demonstrate that classical trust-region methods and splitting techniques (e.g., forward-backward) are special cases of the proposed oracle-based framework.
• To prove that the assumption of a positive definite second-order term in prior convergence proofs is necessary and cannot be removed.

Proposed method

• The method uses cutting plane oracles that generate local models of the objective function via tangents or subgradients at trial points.
• It defines four basic rules for oracles to ensure convergence, including consistency with the objective's subdifferential and proper model construction.
• The trust-region algorithm accumulates cutting planes from unsuccessful (null) steps to refine the working model and guide the next serious iterate.
• A strict model is required for convergence, defined as a model that approximates the objective with sufficient accuracy near the current iterate.
• The framework allows for both model-based oracles and the all-tangents oracle, which includes all subgradients in a neighborhood, ensuring robustness.
• Convergence is proven by showing that all accumulation points of serious iterates are critical points satisfying 0 ∈ ∂f(x*) + N_C(x*).

Experimental results

Research questions

• RQ1 Can global convergence be established for non-smooth trust-region methods using a general class of cutting plane oracles?
• RQ2 Is the method of downshifted tangents globally convergent within the trust-region framework, and does it satisfy the proposed oracle rules?
• RQ3 Can aggregation techniques from convex bundle methods be extended to non-convex trust-region settings without additional assumptions?
• RQ4 Is the assumption of a positive definite second-order term in prior convergence proofs necessary, or can it be removed?
• RQ5 Are classical trust-region methods and splitting techniques like forward-backward splitting special cases of the proposed oracle-based framework?

Key findings

• Global convergence is proven for all trust-region algorithms using cutting plane oracles that satisfy the four defined rules, ensuring all accumulation points are critical points.
• The method of downshifted tangents is formally established as globally convergent in the trust-region framework, resolving a previously open question.
• Aggregation techniques used in convex bundle methods can be extended to non-convex trust regions, but only under the strict model condition.
• The assumption of a positive definite second-order term in earlier convergence proofs is shown to be essential and cannot be removed.
• Classical trust-region methods and forward-backward splitting are shown to be special cases of the proposed framework, with convergence depending on the strictness of the oracle.
• The all-tangents oracle (including all subgradients in a neighborhood) ensures global convergence and subsumes many existing methods under a single theoretical umbrella.