[Paper Review] Accelerating Adaptive Cubic Regularization of Newton's Method via Random Sampling

Xi Chen, Bo Jiang+2 more|arXiv (Cornell University)|Feb 15, 2018

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes SACR, an accelerated adaptive cubic regularization method for large-scale optimization with nonconvex component functions. By using uniform or non-uniform random sampling to approximate the Hessian, SACR achieves a global iteration complexity of $\mathcal{O}(\epsilon^{-1/3})$ with high probability—matching the full-Hessian accelerated method—while maintaining robustness and efficiency on real-world datasets.

ABSTRACT

In this paper, we consider an unconstrained optimization model where the objective is a sum of a large number of possibly nonconvex functions, though overall the objective is assumed to be smooth and convex. Our bid to solving such model uses the framework of cubic regularization of Newton's method. As well known, the crux in cubic regularization is its utilization of the Hessian information, which may be computationally expensive for large-scale problems. To tackle this, we resort to approximating the Hessian matrix via sub-sampling. In particular, we propose to compute an approximated Hessian matrix by either extit{uniformly}\/ or extit{non-uniformly}\/ sub-sampling the components of the objective. Based upon such sampling strategy, we develop accelerated adaptive cubic regularization approaches and provide theoretical guarantees on global iteration complexity of $O(ε^{-1/3})$ with high probability, which matches that of the original accelerated cubic regularization methods \cite{Jiang-2017-Unified} using the extit{full}\/ Hessian information. Interestingly, we show that in the worst case scenario our algorithm still achieves an $O\left(\log(ε^{-1})ε^{-5/6} ight)$ iteration complexity bound. The performances of the proposed methods on the regularized logistic regression problems show a clear effect of acceleration in terms of the epoch counts on several real data sets.

Motivation & Objective

• Address the high computational cost of Hessian computation in cubic regularization Newton methods for large-scale problems.
• Develop an adaptive, parameter-free algorithm that avoids explicit knowledge of Lipschitz constants for gradient and Hessian.
• Achieve accelerated convergence rates comparable to full-Hessian methods while using sub-sampled Hessian approximations.
• Ensure theoretical guarantees on iteration complexity under random sampling, even in worst-case scenarios.
• Demonstrate practical efficiency on real-world regularized logistic regression problems with large-scale datasets.

Proposed method

• Propose a sub-sampled Hessian approximation via uniform or non-uniform sampling of component functions in the finite-sum objective.
• Integrate the sub-sampled Hessian into an adaptive cubic regularization framework to stabilize and accelerate convergence.
• Use a randomized sampling strategy to reduce Hessian computation cost while preserving global convergence guarantees.
• Apply generalized conjugate gradient methods to solve the subproblems efficiently, enabling scalability to high-dimensional problems.
• Design an adaptive stepsize rule that adjusts based on local curvature and sampling variance, eliminating the need for prior knowledge of problem parameters.
• Leverage novel proof techniques to establish high-probability complexity bounds under sampling uncertainty.

Experimental results

Research questions

• RQ1 Can adaptive cubic regularization be accelerated using only sub-sampled Hessian approximations without full Hessian computation?
• RQ2 What is the global iteration complexity of such a sub-sampled accelerated method under high-probability sampling guarantees?
• RQ3 How does the algorithm perform in the worst-case scenario when sampling quality is poor?
• RQ4 Can the method maintain convergence rates matching the full-Hessian accelerated counterpart while being computationally efficient?
• RQ5 Does the algorithm outperform standard stochastic and quasi-Newton methods in practice on large-scale machine learning problems?

Key findings

• SACR achieves a global iteration complexity of $\mathcal{O}(\epsilon^{-1/3})$ with high probability, matching the theoretical rate of the full-Hessian accelerated method.
• In the worst-case scenario, the algorithm still attains an iteration complexity of $\mathcal{O}(\epsilon^{-5/6}\log(\epsilon^{-1}))$, demonstrating robustness.
• Empirical results on eight real datasets show SACR outperforms SGD, LBFGS, mini-batch LBFGS, and ACR in convergence speed and solution accuracy.
• The algorithm is more efficient than its deterministic counterpart (ACR) due to sub-sampling, especially in high-dimensional settings.
• SACR maintains fast convergence even when the Hessian is approximated, and the subproblem solving remains efficient via conjugate gradient methods.
• The proposed proof techniques are novel and may be of independent interest for analyzing randomized second-order methods.