[Paper Review] DynaNewton - Accelerating Newton's Method for Machine Learning

Hadi Daneshmand, Aurélien Lucchi+1 more|arXiv (Cornell University)|May 20, 2016

Sparse and Compressive Sensing Techniques15 references4 citations

TL;DR

DynaNewton accelerates Newton's method for machine learning by dynamically adjusting sample size and regularization strength via a continuation path, ensuring each Newton iteration starts within the quadratic convergence region. This leads to super-linear convergence and significant speedups—often achieving near-optimal solutions in under 2 effective epochs across diverse datasets.

ABSTRACT

Newton's method is a fundamental technique in optimization with quadratic convergence within a neighborhood around the optimum. However reaching this neighborhood is often slow and dominates the computational costs. We exploit two properties specific to empirical risk minimization problems to accelerate Newton's method, namely, subsampling training data and increasing strong convexity through regularization. We propose a novel continuation method, where we define a family of objectives over increasing sample sizes and with decreasing regularization strength. Solutions on this path are tracked such that the minimizer of the previous objective is guaranteed to be within the quadratic convergence region of the next objective to be optimized. Thereby every Newton iteration is guaranteed to achieve super-linear contractions with regard to the chosen objective, which becomes a moving target. We provide a theoretical analysis that motivates our algorithm, called DynaNewton, and characterizes its speed of convergence. Experiments on a wide range of data sets and problems consistently confirm the predicted computational savings.

Motivation & Objective

• Address the slow initial phase of Newton's method, where most computation is spent before reaching quadratic convergence.
• Overcome the high per-iteration cost of batch Newton methods by subsampling training data and reducing regularization over time.
• Develop a continuation method that tracks solutions across progressively larger samples and weaker regularization, ensuring each step starts in the quadratic convergence region.
• Improve robustness to poor initialization by leveraging previous solutions as starting points for subsequent subproblems.
• Achieve faster convergence on both empirical and expected risk, with theoretical and empirical validation of computational savings.

Proposed method

• Define a family of objectives $ f^{ ext{S}}_{\nu}(\mathbf{x}) = \frac{1}{|\mathcal{S}|} \sum_{\mathbf{z} \in \mathcal{S}} \phi^{\mathbf{z}}(\mathbf{x}) + \nu \Omega(\mathbf{x}) $, where $ \nu \propto 1/n $, with $ n $ being the sample size.
• Use a dynamic continuation path: increase sample size $ n $ and decrease regularization $ \nu $ in a controlled manner, ensuring the solution of one subproblem lies within the quadratic convergence region of the next.
• Apply a data-adaptive strategy to determine the increment factor $ \alpha $, which controls the rate of sample size increase and regularization decrease.
• Leverage the solution from a previous subproblem as the initial point for the next Newton iteration, ensuring fast convergence.
• Integrate with quasi-Newton methods like L-BFGS by applying the same continuation framework, extending the approach beyond exact Hessian computation.
• Theoretical analysis ensures super-linear contraction per Newton step by maintaining the initial point within the quadratic convergence ball of each subsequent problem.

Experimental results

Research questions

• RQ1 Can dynamic subsampling and adaptive regularization be used to accelerate Newton's method in large-scale machine learning?
• RQ2 Does a continuation path that increases sample size and decreases regularization guarantee that each Newton step starts within the quadratic convergence region?
• RQ3 How does DynaNewton compare to standard Newton’s method, SAGA, and L-BFGS in terms of convergence speed on empirical and expected risk?
• RQ4 To what extent does the algorithm remain robust to poor initialization compared to standard Newton’s method?
• RQ5 Can the continuation framework be extended to quasi-Newton methods like L-BFGS while preserving convergence speed and accuracy?

Key findings

• DynaNewton achieves super-linear convergence by ensuring each Newton iteration starts within the quadratic convergence region of the next subproblem.
• On all tested datasets (a9a, w8a, covtype, susy), DynaNewton reaches near-optimal solutions in less than 2 effective epochs, significantly outperforming standard Newton’s method.
• The algorithm reduces computational cost by dynamically adapting sample size and regularization, with empirical results showing substantial speedups—especially on expected risk minimization.
• The data-adaptive strategy for choosing the increment factor $ \alpha $ ensures stable convergence, avoiding divergence seen with fixed small $ \alpha $ values.
• DynaNewton is robust to poor initialization: unlike standard Newton’s method, convergence is not significantly slowed by distant starting points.
• Empirical results confirm that the continuation method extends to L-BFGS, suggesting broader applicability beyond exact Hessian methods, though theoretical guarantees remain open for quasi-Newton variants.