[Paper Review] A Progressive Batching L-BFGS Method for Machine Learning

Raghu Bollapragada, Dheevatsa Mudigere+3 more|arXiv (Cornell University)|Feb 15, 2018

Stochastic Gradient Optimization Techniques40 references57 citations

TL;DR

Introduces PBQN, a progressive-batching quasi-Newton algorithm that combines progressive sampling, a stochastic Armijo-style line search, and stable L-BFGS updating to train logistic regression and deep nets with good generalization and parallelism. Convergence theory is provided for both convex and nonconvex settings.

ABSTRACT

The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadratic models of the objective function. All of this appears to call for a full batch approach, but since small batch sizes give rise to faster algorithms with better generalization properties, L-BFGS is currently not considered an algorithm of choice for large-scale machine learning applications. One need not, however, choose between the two extremes represented by the full batch or highly stochastic regimes, and may instead follow a progressive batching approach in which the sample size increases during the course of the optimization. In this paper, we present a new version of the L-BFGS algorithm that combines three basic components - progressive batching, a stochastic line search, and stable quasi-Newton updating - and that performs well on training logistic regression and deep neural networks. We provide supporting convergence theory for the method.

Motivation & Objective

• Motivate the use of progressive batching to blend the speed of stochastic methods with the accuracy of full-batch quasi-Newton approaches.
• Develop a PBQN algorithm that integrates progressive sampling, a stochastic line search, and stable L-BFGS updating.
• Provide convergence theory for strongly convex and nonconvex objectives under progressive sampling.
• Demonstrate the method on logistic regression and neural networks to assess performance and generalization.
• Discuss practical aspects and potential for parallel/distributed implementation.

Proposed method

• Propose PBQN: x_{k+1} = x_{k} - α_{k} H_{k} g_{k}^{S_{k}} with H_{k} from L-BFGS and g_{k}^{S_{k}} as a subsampled gradient.
• Use a progressive sampling scheme where the batch size |S_{k}| increases when a stochastic inner-product quasi-Newton (IPQN) test (based on the angle with the true direction) is not satisfied.
• Extend inner-product tests to second-order methods to ensure the search direction aligns with the true quasi-Newton direction with high probability.
• Introduce an Armijo-style backtracking line search with an initial step α_{k} guided by a variance-informed estimate (Eq. 14) to ensure decrease in expectation.
• Employ an overlap-based strategy (y_{k} = g_{k+1}^{O_{k}} - g_{k}^{O_{k}}) to stabilize Hessian updates when batches change between iterations.
• Provide two options for y_{k} computation: multi-batch (MB) with 25% overlap and full-overlap (FO) approach, and use a curvature condition y_{k}^{T}s_{k} > ε‖s_{k}‖^{2} for update eligibility.

Experimental results

Research questions

• RQ1 Can progressive batching with stochastic line search provide reliable descent directions for quasi-Newton updates in ML settings?
• RQ2 Do PBQN methods achieve competitive training and generalization performance on logistic regression and deep neural networks compared with SG-based methods?
• RQ3 What convergence guarantees hold for PBQN in strongly convex and nonconvex regimes under progressive sampling?
• RQ4 How does batch overlap choice for curvature pairs affect stability and performance?
• RQ5 Is PBQN amenable to parallel/distributed implementations with good scalability?

Key findings

• PBQN competes well with SG and SVRG on logistic regression in terms of training error, test loss, and test accuracy across datasets.
• The MB curvature vector approach generally outperforms FO in logistic regression experiments.
• In neural networks, PBQN attains similar best test accuracies to SG and Adam, with fewer iterations but more gradient evaluations.
• The initial step length from the variance-based formula (Eq. 14) is reliably accepted by the line search, leading to efficient backtracking.
• Convergence theory shows linear convergence for strongly convex F under a suitable step size, and gradient-norm convergence for nonconvex F under standard smoothness and boundedness assumptions.
• The method requires almost no parameter tuning due to second-order information and demonstrates robustness across tested problems.