[Paper Review] Revisiting Sub-sampled Newton Methods

Haishan Ye, Luo Luo+1 more|arXiv (Cornell University)|Aug 8, 2016

Stochastic Gradient Optimization Techniques19 references3 citations

TL;DR

This paper introduces two novel sub-sampled Newton methods—Refined Sub-sampled Newton (ReSubNewton) and Refined Sketch Newton (ReSkeNewton)—that achieve superlinear and quadratic convergence without increasing sampling size, even for ill-conditioned Hessians. The methods leverage refined Hessian approximation and sketching techniques, demonstrating theoretical convergence improvements and empirical efficiency gains over existing sub-sampled Newton approaches.

ABSTRACT

Many machine learning models depend on solving a large scale optimization problem. Recently, sub-sampled Newton methods have emerged to attract much attention for optimization due to their efficiency at each iteration, rectified a weakness in the ordinary Newton method of suffering a high cost at each iteration while commanding a high convergence rate. In this work we propose two new efficient Newton-type methods, Refined Sub-sampled Newton and Refined Sketch Newton. Our methods exhibit a great advantage over existing sub-sampled Newton methods, especially when Hessian-vector multiplication can be calculated efficiently. Specifically, the proposed methods are shown to converge superlinearly in general case and quadratically under a little stronger assumption. The proposed methods can be generalized to a unifying framework for the convergence proof of several existing sub-sampled Newton methods, revealing new convergence properties. Finally, we empirically evaluate the performance of our methods on several standard datasets and the results show consistent improvement in computational efficiency.

Motivation & Objective

• Address the open problem of achieving superlinear and quadratic convergence in sub-sampled Newton methods without increasing sampling complexity.
• Develop a unifying theoretical framework to analyze convergence properties of existing sub-sampled Newton methods.
• Investigate whether the Lipschitz continuity condition is necessary for convergence, particularly in non-Lipschitz settings.
• Enhance computational efficiency and convergence rates for large-scale machine learning optimization problems with ill-conditioned Hessians.

Proposed method

• Propose Refined Sub-sampled Newton (ReSubNewton), which refines the Hessian approximation using a refined sampling strategy to improve curvature estimation.
• Introduce Refined Sketch Newton (ReSkeNewton), which applies sketching techniques with improved subspace embedding guarantees to approximate the Hessian efficiently.
• Utilize refined error bounds on Hessian approximation via $ abla^2F(x^{(t)}) - H^{(t)}$ and $ orm{[H^{(t)}]^{-1}}$ to ensure convergence under weaker assumptions.
• Establish a unifying convergence framework based on a key theorem (Theorem 7) that bounds the convergence rate via $ ho = orm{( abla^2F - H^{(t)})} orm{[H^{(t)}]^{-1}}$.
• Apply subspace embedding theory to ensure that the Hessian approximation $H^{(t)}$ satisfies $(1- ho) abla^2F rianglelefteq H^{(t)} rianglelefteq (1+ ho) abla^2F$ with high probability.
• Integrate sketching matrices $S$ that are $( ho rac{ au}{K})$-subspace embeddings to control approximation error in the Hessian and its inverse.

Experimental results

Research questions

• RQ1 Can sub-sampled Newton methods achieve superlinear or quadratic convergence without increasing the number of samples?
• RQ2 Is there a unifying theoretical framework to analyze the convergence of various sub-sampled Newton methods?
• RQ3 Is the Lipschitz continuity condition on the Hessian necessary for convergence in sub-sampled Newton methods?
• RQ4 How do refined Hessian approximations improve convergence in ill-conditioned settings compared to standard sub-sampled Newton methods?

Key findings

• ReSubNewton and ReSkeNewton achieve superlinear convergence in the general case without increasing sampling size, resolving a key open problem in sub-sampled Newton methods.
• Under a slightly stronger assumption, the proposed methods achieve quadratic convergence, which is the first such result for sub-sampled Newton methods.
• The methods are shown to converge linearly with probability $(1- ho)^T$ when $ ho < 1/2$, and superlinearly when $ ho o 0$ as iterations progress.
• Theoretical analysis reveals that the convergence rate depends on $ ho = orm{( abla^2F - H^{(t)})} orm{[H^{(t)}]^{-1}}$, and the proposed methods keep this ratio small through refined Hessian estimation.
• Empirical evaluation on standard datasets confirms consistent improvements in computational efficiency over existing sub-sampled Newton methods.
• The framework unifies the convergence analysis of multiple existing sub-sampled Newton methods, revealing new convergence properties such as linear-quadratic rates under Lipschitz conditions.