[Paper Review] On the Quadratic Convergence of the Cubic Regularization Method under a Local Error Bound Condition

Man–Chung Yue, Zirui Zhou+1 more|arXiv (Cornell University)|Jan 29, 2018

Sparse and Compressive Sensing Techniques22 references3 citations

TL;DR

This paper establishes Q-quadratic convergence of the cubic regularization (CR) method under a local error bound (EB) condition, a significantly weaker requirement than the classical non-degeneracy condition. The authors prove that the CR method converges quadratically to second-order critical points—even in degenerate, non-isolated minimizer settings—by leveraging the EB condition, which they further show is equivalent to a quadratic growth condition without convexity assumptions.

ABSTRACT

In this paper we consider the cubic regularization (CR) method for minimizing a twice continuously differentiable function. While the CR method is widely recognized as a globally convergent variant of Newton's method with superior iteration complexity, existing results on its local quadratic convergence require a stringent non-degeneracy condition. We prove that under a local error bound (EB) condition, which is much weaker a requirement than the existing non-degeneracy condition, the sequence of iterates generated by the CR method converges at least Q-quadratically to a second-order critical point. This indicates that adding a cubic regularization not only equips Newton's method with remarkable global convergence properties but also enables it to converge quadratically even in the presence of degenerate solutions. As a byproduct, we show that without assuming convexity, the proposed EB condition is equivalent to a quadratic growth condition, which could be of independent interest. To demonstrate the usefulness and relevance of our convergence analysis, we focus on two concrete nonconvex optimization problems that arise in phase retrieval and low-rank matrix recovery, respectively, and prove that with overwhelming probability, the sequence of iterates generated by the CR method for solving these two problems converges at least Q-quadratically to a global minimizer. We also present numerical results of the CR method when applied to solve these two problems to support and complement our theoretical development.

Motivation & Objective

• To establish local quadratic convergence of the cubic regularization (CR) method under a weaker condition than the standard non-degeneracy assumption.
• To analyze the convergence behavior of the CR method in the presence of non-isolated minimizers, which are common in nonconvex problems such as phase retrieval and low-rank matrix recovery.
• To demonstrate that the proposed local error bound (EB) condition is equivalent to a quadratic growth condition without assuming convexity, a result of independent interest.
• To validate the theoretical findings through numerical experiments on phase retrieval and low-rank matrix recovery problems.
• To show that with overwhelming probability, the CR method converges Q-quadratically to a global minimizer in two concrete nonconvex optimization problems.

Proposed method

• The authors introduce a local error bound (EB) condition that characterizes the local geometry of the objective function near second-order critical points.
• They prove that under this EB condition, the sequence of iterates generated by the CR method converges at least Q-quadratically to a second-order critical point.
• The analysis relies on the fact that any accumulation point of the CR iterates is a second-order critical point, which is a key structural property of the method.
• The authors establish the equivalence between the proposed EB condition and a quadratic growth condition in the nonconvex setting, without requiring convexity.
• They apply the theory to two nonconvex problems—phase retrieval and low-rank matrix recovery—by showing that the EB condition holds with overwhelming probability under standard random model assumptions.
• Numerical experiments are conducted using Algorithm 1 (CR method) with adaptive regularization, where convergence is monitored via relative error (RE) and termination is triggered when RE < 10⁻⁸.

Experimental results

Research questions

• RQ1 Can the cubic regularization method achieve Q-quadratic convergence under a condition weaker than the standard non-degeneracy assumption?
• RQ2 Is the local error bound (EB) condition equivalent to a quadratic growth condition in nonconvex optimization without assuming convexity?
• RQ3 Does the CR method converge quadratically to a global minimizer in nonconvex problems like phase retrieval and low-rank matrix recovery, under typical random model assumptions?
• RQ4 How does the convergence behavior of the CR method manifest in practice for these nonconvex problems, and does it align with the theoretical predictions?
• RQ5 Can the CR method maintain fast local convergence even when the Hessian is degenerate or the minimizer is non-isolated?

Key findings

• The CR method converges at least Q-quadratically to a second-order critical point under a local error bound (EB) condition, which is strictly weaker than the non-degeneracy condition.
• The EB condition is equivalent to a quadratic growth condition in the nonconvex setting, a result that holds without assuming convexity.
• For phase retrieval, with overwhelming probability, the sequence of iterates generated by the CR method converges Q-quadratically to a global minimizer.
• For low-rank matrix recovery, under the same probabilistic model, the CR method also achieves Q-quadratic convergence to a global minimizer with overwhelming probability.
• Numerical results confirm the theoretical convergence rate: the relative error (RE) decays superlinearly in the final iterations, indicating at least quadratic convergence.
• The time to reach a solution scales reasonably with problem size, with runtimes of 4.7s (n=64), 12.3s (n=128), 129.6s (n=256), and 577.6s (n=512) for phase retrieval, and up to 5017.6s for larger low-rank recovery problems.