[Paper Review] Nestrov's Acceleration For Second Order Method

Haishan Ye, Zhihua Zhang|arXiv (Cornell University)|May 19, 2017

Stochastic Gradient Optimization Techniques2 references3 citations

TL;DR

This paper proposes an accelerated regularized sub-sampled Newton method by applying Nesterov's acceleration technique to approximate Newton methods, significantly improving convergence speed and performance. Theoretical analysis confirms acceleration benefits similar to first-order methods, and experiments show superior empirical results compared to baseline and state-of-the-art algorithms.

ABSTRACT

Optimization plays a key role in machine learning. Recently, stochastic second-order methods have attracted much attention due to their low computational cost in each iteration. However, these algorithms might perform poorly especially if it is hard to approximate the Hessian well and efficiently. As far as we know, there is no effective way to handle this problem. In this paper, we resort to Nestrov's acceleration technique to improve the convergence performance of a class of second-order methods called approximate Newton. We give a theoretical analysis that Nestrov's acceleration technique can improve the convergence performance for approximate Newton just like for first-order methods. We accordingly propose an accelerated regularized sub-sampled Newton. Our accelerated algorithm performs much better than the original regularized sub-sampled Newton in experiments, which validates our theory empirically. Besides, the accelerated regularized sub-sampled Newton has good performance comparable to or even better than state-of-art algorithms.

Motivation & Objective

• To address the poor performance of stochastic second-order methods when Hessian approximation is inaccurate.
• To explore whether Nesterov's acceleration, effective in first-order methods, can enhance second-order methods like approximate Newton.
• To develop a theoretically grounded, accelerated variant of regularized sub-sampled Newton for improved convergence.

Proposed method

• Adapt Nesterov's acceleration framework to the approximate Newton method, introducing momentum-like updates with extrapolation.
• Integrate regularization into the sub-sampled Hessian approximation to stabilize optimization in ill-conditioned settings.
• Use a two-loop update scheme: one for extrapolated search points and one for Hessian-based descent steps.
• Apply Nesterov's optimal acceleration strategy to the sub-sampled Hessian update, ensuring convergence rate improvements.

Experimental results

Research questions

• RQ1 Can Nesterov's acceleration technique improve the convergence of approximate Newton methods, similar to its success in first-order methods?
• RQ2 How does the accelerated regularized sub-sampled Newton perform compared to the original regularized sub-sampled Newton in practice?
• RQ3 Does the proposed method achieve performance comparable to or better than state-of-the-art optimization algorithms?

Key findings

• The proposed accelerated regularized sub-sampled Newton achieves faster convergence than the original regularized sub-sampled Newton in all tested experiments.
• Theoretical analysis confirms that Nesterov's acceleration improves convergence rates for approximate Newton methods, just as in first-order settings.
• Empirical results show the accelerated method outperforms or matches state-of-the-art algorithms across multiple benchmark datasets.
• The method demonstrates robustness even when Hessian approximation is challenging, due to the acceleration mechanism.