[Paper Review] Parallel Stochastic Gradient Descent with Sound Combiners

Saeed Maleki, Madanlal Musuvathi+1 more|arXiv (Cornell University)|May 22, 2017

Stochastic Gradient Optimization Techniques12 references3 citations

TL;DR

This paper proposes SymSGD, a parallel stochastic gradient descent algorithm that preserves the sequential semantics of SGD by using sound model combiners to accurately combine local models across threads. It achieves up to 11× speedup over sequential SGD on 16 cores and 2.2× faster average performance than Hogwild! while maintaining equivalent accuracy by minimizing staleness and communication overhead through first-order approximation of gradient updates.

ABSTRACT

Stochastic gradient descent (SGD) is a well known method for regression and classification tasks. However, it is an inherently sequential algorithm at each step, the processing of the current example depends on the parameters learned from the previous examples. Prior approaches to parallelizing linear learners using SGD, such as HOGWILD! and ALLREDUCE, do not honor these dependencies across threads and thus can potentially suffer poor convergence rates and/or poor scalability. This paper proposes SYMSGD, a parallel SGD algorithm that, to a first-order approximation, retains the sequential semantics of SGD. Each thread learns a local model in addition to a model combiner, which allows local models to be combined to produce the same result as what a sequential SGD would have produced. This paper evaluates SYMSGD's accuracy and performance on 6 datasets on a shared-memory machine shows upto 11x speedup over our heavily optimized sequential baseline on 16 cores and 2.2x, on average, faster than HOGWILD!.

Motivation & Objective

• To address the scalability and convergence issues of existing parallel SGD methods like Hogwild! and AllReduce, which violate sequential dependencies due to stale or infrequent updates.
• To enable efficient, high-fidelity parallelization of SGD on shared-memory systems by preserving the first-order behavior of sequential SGD.
• To reduce communication and cache-coherence overhead in multi-core environments, especially across multiple sockets, where existing methods like Hogwild! suffer from scalability limits.
• To design a model combiner mechanism that accurately captures the effect of intermediate global model updates on local models, ensuring convergence fidelity.
• To demonstrate that parallel SGD can achieve near-sequential accuracy with significantly improved performance on modern multi-core machines with large memory capacity.

Proposed method

• Each thread maintains a local model and a model combiner that captures the first-order effect of gradient updates on the global model during the interval between synchronization points.
• The model combiner is computed using the Hessian and gradient information to approximate how the global model would have changed had it been updated sequentially.
• Periodically, threads synchronize by applying their local model updates to the global model using the model combiner to correct for any intermediate changes in the global model.
• The algorithm supports both async and multi-round (MR) synchronization modes, with the async variant minimizing communication overhead by targeting only frequently accessed features.
• The method is designed to be generalizable to various machine learning problems and parallel architectures, with evaluation focused on linear models in shared-memory systems.
• The model combiner ensures that the final global model is equivalent to what a sequential SGD would produce, up to first-order approximation, by accounting for the temporal evolution of the global model.

Experimental results

Research questions

• RQ1 Can a parallel SGD algorithm preserve the convergence behavior of sequential SGD while achieving high scalability on multi-core systems?
• RQ2 How can model combiners be constructed to accurately approximate the first-order effects of sequential gradient updates in a parallel setting?
• RQ3 Does reducing communication frequency in parallel SGD lead to better performance and scalability compared to approaches like Hogwild! that communicate after every example?
• RQ4 Can the proposed method maintain high accuracy on both sparse and dense datasets, unlike Hogwild! which is optimized for sparse data?
• RQ5 To what extent does the use of model combiners improve performance and scalability across multiple sockets in shared-memory systems?

Key findings

• SymSGD achieves up to 11× speedup over a heavily optimized sequential SGD baseline on 16 cores, demonstrating strong scalability on multi-core machines.
• On average, SymSGD is 2.2× faster than Hogwild! across the evaluated datasets, with consistent performance improvements even at scale.
• Async-SymSGD scales linearly across multiple sockets, unlike Hogwild!, which shows reduced performance from 8 to 16 cores due to cache-coherence overhead.
• The model combiner mechanism successfully approximates sequential SGD behavior, ensuring that the final model matches the accuracy of the sequential baseline.
• MR-SymSGD and Async-SymSGD both outperform Hogwild! in multi-socket configurations, with Async-SymSGD being faster due to reduced communication overhead.
• The method maintains high accuracy on both sparse and dense datasets, unlike Hogwild! which is less effective on dense data and suffers from staleness in AllReduce-style approaches.