[Paper Review] Slow Learners are Fast

John Langford, Alexander J. Smola+1 more|ArXiv.org|Nov 3, 2009

Advanced Bandit Algorithms Research12 references202 citations

TL;DR

This paper proposes parallel online learning algorithms that use delayed parameter updates to overcome the sequential bottleneck of traditional online methods, demonstrating theoretically and empirically that convergence is preserved even with significant delays. The key contribution is showing that slow learners (due to delayed updates) can still achieve fast, scalable learning on large-scale problems by leveraging parallelism across multiple cores.

ABSTRACT

Online learning algorithms have impressive convergence properties when it comes to risk minimization and convex games on very large problems. However, they are inherently sequential in their design which prevents them from taking advantage of modern multi-core architectures. In this paper we prove that online learning with delayed updates converges well, thereby facilitating parallel online learning.

Motivation & Objective

• Address the sequential bottleneck in online learning algorithms that limits scalability on modern multi-core architectures.
• Overcome the I/O and CPU bottlenecks in large-scale learning by enabling asynchronous, parallel updates.
• Develop theoretically grounded algorithms that maintain convergence despite delayed parameter updates.
• Demonstrate practical scalability and performance gains in real-world and synthetic experiments with varying problem complexity.

Proposed method

• Use asynchronous stochastic gradient descent (SGD) across multiple cores, where each core computes gradients independently and updates a shared global parameter vector with delay.
• Implement a data-parallel architecture where feature space is partitioned among slave threads, each computing partial dot products, and a master aggregates and applies updates.
• Apply lazy regularization and learning rate decay ($\eta_t = 1/\sqrt{t}$) to maintain stability and reduce memory pressure.
• Use feature hashing to manage high-dimensional input spaces efficiently, with $2^{18}$ and $2^{24}$ bins for different datasets.
• Limit maximum delay to 100 examples to prevent pipeline stalls and ensure robustness.
• Simulate delayed updates in controlled settings to analyze convergence behavior under increasing delay ($\tau \in \{0,10,100,1000\}$).

Experimental results

Research questions

• RQ1 Can online learning algorithms maintain convergence when updates are delayed due to parallel execution?
• RQ2 How does the magnitude of delay affect learning performance in practice, especially in high-dimensional or complex feature spaces?
• RQ3 To what extent can parallelization improve training speed on computationally intensive problems compared to sequential processing?
• RQ4 Does the benefit of delayed updates vary with problem complexity—e.g., linear vs. quadratic feature representations?
• RQ5 Can theoretical convergence guarantees be extended to practical, asynchronous, multi-core implementations of online learning?

Key findings

• Small delays (up to 100 examples) cause negligible degradation in performance on linear feature problems, with convergence preserved.
• Performance degrades significantly only under very large delays (e.g., 1000 examples), particularly in simpler problems like email classification.
• In harder problems with complex representations (e.g., quadratic features), parallelization with delayed updates leads to dramatic speedups, as single-example computation time exceeds 1ms.
• The experimental results confirm that delayed updates do not incur a proportional penalty in convergence, supporting the theoretical claim that independence between examples reduces the effective impact of delay.
• Real-world parallelization of the algorithm was feasible and scalable, especially on memory- and compute-bound problems, with over 150,000 examples/sec in serial mode, but far greater gains in complex settings.
• The use of feature hashing with $2^{18}$ and $2^{24}$ bins preserved model quality, validating its use in high-dimensional, large-scale learning.