[Paper Review] Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods

Jascha Sohl‐Dickstein, Ben Poole+1 more|arXiv (Cornell University)|Nov 9, 2013

Stochastic Gradient Optimization TechniquesComputer Science42 references83 citations

TL;DR

This paper proposes Stochastic First-Order (SFO), a novel optimization algorithm that unifies stochastic gradient descent (SGD) and quasi-Newton methods by maintaining independent Hessian approximations for each subfunction in a sum-structured objective. By projecting these approximations into a low-dimensional, adaptive subspace, SFO achieves fast convergence with minimal hyperparameter tuning, outperforming both SGD and Hessian-free methods on diverse machine learning tasks including deep networks and logistic regression.

ABSTRACT

We present an algorithm for minimizing a sum of functions that combines the computational efficiency of stochastic gradient descent (SGD) with the second order curvature information leveraged by quasi-Newton methods. We unify these disparate approaches by maintaining an independent Hessian approximation for each contributing function in the sum. We maintain computational tractability and limit memory requirements even for high dimensional optimization problems by storing and manipulating these quadratic approximations in a shared, time evolving, low dimensional subspace. Each update step requires only a single contributing function or minibatch evaluation (as in SGD), and each step is scaled using an approximate inverse Hessian and little to no adjustment of hyperparameters is required (as is typical for quasi-Newton methods). This algorithm contrasts with earlier stochastic second order techniques that treat the Hessian of each contributing function as a noisy approximation to the full Hessian, rather than as a target for direct estimation. We experimentally demonstrate improved convergence on seven diverse optimization problems. The algorithm is released as open source Python and MATLAB packages.

Motivation & Objective

• To address the computational inefficiency of full quasi-Newton methods on large-scale optimization problems by combining the efficiency of stochastic gradient descent with second-order curvature information.
• To overcome the limitations of existing stochastic second-order methods that treat subfunction Hessians as noisy approximations to the full Hessian, rather than as direct targets for estimation.
• To reduce hyperparameter sensitivity common in quasi-Newton methods while maintaining fast convergence through adaptive Hessian approximation.
• To enable scalable, low-memory optimization in high-dimensional settings by projecting Hessian approximations into a time-evolving, low-dimensional subspace.

Proposed method

• The algorithm maintains a separate quadratic approximation $ g_i^t( extbf{x}) $ for each subfunction $ f_i( extbf{x}) $, using BFGS updates to refine these approximations based on gradient history.
• Optimization proceeds in two alternating steps: (1) minimizing the sum of current approximations $ G^{t-1}( extbf{x}) = \sum_i g_i^{t-1}( extbf{x}) $, and (2) updating one $ g_i^t( extbf{x}) $ using a second-order expansion around the new parameter value $ \textbf{x}^t $.
• All Hessian approximations are stored and manipulated in a shared, low-dimensional subspace spanned by the history of gradients and parameter updates, ensuring computational and memory tractability.
• The method uses only a single subfunction evaluation per step (like SGD), yet scales steps using approximate inverse Hessian information (like quasi-Newton), reducing the need for manual hyperparameter tuning.
• The subspace is adaptively updated over time, capturing dominant curvature directions without requiring full Hessian computation.
• The algorithm is designed to be naturally parallelizable, allowing asynchronous updates of individual subfunction approximations and support for online/infinite data settings.

Experimental results

Research questions

• RQ1 Can a stochastic optimization method effectively leverage second-order curvature information without incurring the computational cost of full Hessian computation?
• RQ2 Can independent Hessian approximations for each subfunction be maintained efficiently in a shared low-dimensional subspace to enable scalable optimization?
• RQ3 Does combining the step-size adaptivity of quasi-Newton methods with the computational efficiency of SGD lead to faster convergence on large-scale problems?
• RQ4 Can this approach reduce or eliminate the need for manual hyperparameter tuning common in both SGD and quasi-Newton methods?

Key findings

• SFO achieved faster convergence than Hessian-free optimization on a twelve-layer neural network trained on the CURVES dataset, requiring approximately one-tenth the number of effective data passes.
• On seven diverse optimization problems, including logistic regression, Ising models, and deep networks, SFO demonstrated consistently improved convergence compared to baseline methods.
• The algorithm required little to no hyperparameter tuning, leveraging curvature information from the data itself to adapt step sizes and directions.
• The use of a low-dimensional subspace enabled memory and computational efficiency even in high-dimensional problems, maintaining scalability.
• The method was successfully released as open-source Python and MATLAB packages, supporting practical deployment in machine learning applications.
• Empirical results show that SFO outperforms both standard SGD and quasi-Newton methods in terms of convergence speed and robustness across a wide range of tasks.