[Paper Review] Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent

Wei Xu|arXiv (Cornell University)|Jul 13, 2011

Stochastic Gradient Optimization TechniquesComputer Science17 references120 citations

TL;DR

This paper proposes a learning rate schedule for Averaged Stochastic Gradient Descent (ASGD) that enables optimal convergence in a single pass over large-scale datasets, significantly reducing the number of samples needed to reach asymptotic performance. The method achieves state-of-the-art accuracy on linear classifiers with minimal computational overhead, outperforming standard SGD and other optimization algorithms in both test error and training efficiency.

ABSTRACT

For large scale learning problems, it is desirable if we can obtain the optimal model parameters by going through the data in only one pass. Polyak and Juditsky (1992) showed that asymptotically the test performance of the simple average of the parameters obtained by stochastic gradient descent (SGD) is as good as that of the parameters which minimize the empirical cost. However, to our knowledge, despite its optimal asymptotic convergence rate, averaged SGD (ASGD) received little attention in recent research on large scale learning. One possible reason is that it may take a prohibitively large number of training samples for ASGD to reach its asymptotic region for most real problems. In this paper, we present a finite sample analysis for the method of Polyak and Juditsky (1992). Our analysis shows that it indeed usually takes a huge number of samples for ASGD to reach its asymptotic region for improperly chosen learning rate. More importantly, based on our analysis, we propose a simple way to properly set learning rate so that it takes a reasonable amount of data for ASGD to reach its asymptotic region. We compare ASGD using our proposed learning rate with other well known algorithms for training large scale linear classifiers. The experiments clearly show the superiority of ASGD.

Motivation & Objective

• To address the lack of practical adoption of Averaged Stochastic Gradient Descent (ASGD) in large-scale learning despite its optimal asymptotic convergence.
• To identify why ASGD typically requires an infeasibly large number of samples to reach its asymptotic region under standard learning rate choices.
• To develop a finite-sample analysis of ASGD to guide the design of a learning rate schedule that accelerates convergence to the optimal region.
• To empirically validate that the proposed learning rate schedule enables ASGD to achieve superior test accuracy with only one pass through the data.
• To demonstrate that ASGD with the new learning rate schedule outperforms other state-of-the-art algorithms in terms of accuracy, robustness, and training speed.

Proposed method

• Proposes a finite-sample analysis of Polyak and Juditsky's ASGD method to quantify the number of samples required to reach the asymptotic convergence regime.
• Derives a data-dependent learning rate schedule that ensures ASGD reaches its optimal performance region in a reasonable number of samples, based on the smallest eigenvalue of the Hessian and regularization parameter.
• Uses the running average of SGD parameters, $\bar{\theta}_t = \frac{1}{t}\sum_{j=1}^{t}\theta_j$, as the final model estimator, leveraging its superior asymptotic properties.
• Applies the proposed learning rate to linear models with convex, smooth loss functions (e.g., squared hinge loss) and L2 regularization.
• Employs a practical learning rate schedule $\gamma_t = \frac{\gamma_0}{1 + \gamma_0 \lambda_0 t}$, where $\lambda_0$ is approximated by the regularization parameter $\lambda$.
• Validates the method on both synthetic and real-world datasets, including MNIST, RCV1, and large-scale text and image classification tasks.

Experimental results

Research questions

• RQ1 Why has Averaged Stochastic Gradient Descent (ASGD) received limited attention in recent large-scale learning research despite its optimal asymptotic convergence?
• RQ2 How many training samples are typically required for ASGD to reach its asymptotic performance region under standard learning rate schedules?
• RQ3 Can a finite-sample analysis be used to design a learning rate schedule that accelerates ASGD convergence to the optimal region with fewer samples?
• RQ4 Does ASGD with the proposed learning rate schedule outperform other state-of-the-art optimization algorithms in terms of test accuracy and training efficiency?
• RQ5 Can ASGD achieve strong performance even with non-smooth loss functions like hinge loss, despite theoretical assumptions requiring smoothness?

Key findings

• The proposed learning rate schedule enables ASGD to reach its asymptotic performance region in a reasonable number of samples, significantly reducing the data requirement compared to standard learning rates.
• ASGD with the new learning rate schedule achieves the lowest test error rate across all evaluated datasets, including MNIST9, RCV1, and synthetic data, often outperforming other algorithms after just one data pass.
• ASGD demonstrates superior robustness, with more stable and less volatile performance curves compared to other methods like SGD, SGDQN, and oLBFGS.
• The method achieves state-of-the-art accuracy with only one pass through the data, while other algorithms still show poor performance at the same point.
• Despite theoretical assumptions requiring smooth loss functions, ASGD with the proposed learning rate performs well even with non-smooth hinge loss, indicating broader applicability.
• Training time per pass is comparable to other methods, meaning ASGD reaches high accuracy faster than competing algorithms, especially when considering the one-pass requirement.