[Paper Review] Deep learning with Elastic Averaging SGD

Sixin Zhang, Anna Choromanska+1 more|arXiv (Cornell University)|Dec 20, 2014

Stochastic Gradient Optimization Techniques28 references67 citations

TL;DR

This paper proposes Elastic Averaging SGD (EASGD), a communication-efficient stochastic optimization algorithm for deep learning in distributed settings. By introducing an elastic force linking local worker parameters to a central variable, EASGD enables greater exploration of the loss landscape, leading to faster convergence and improved generalization—demonstrated by state-of-the-art speedups on CIFAR-10 and ImageNet with reduced communication overhead.

ABSTRACT

We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

Motivation & Objective

• To address communication bottlenecks in distributed deep learning training under limited bandwidth.
• To improve optimization performance by enabling greater exploration of the loss landscape through parameter elasticity.
• To design a stable, scalable algorithm that outperforms existing methods like DOWNPOUR and ADMM in both speed and communication efficiency.
• To provide theoretical stability guarantees for the asynchronous variant of the algorithm.
• To demonstrate practical effectiveness on large-scale image classification benchmarks such as CIFAR-10 and ImageNet.

Proposed method

• EASGD introduces a center variable (master) that coordinates local workers via an elastic force, modeled as a quadratic penalty term in the objective function.
• The algorithm uses a dual update rule: local workers update their parameters based on stochastic gradients and deviation from the center, while the center is updated as a weighted moving average of local deviations.
• The core update equations are: $ x^{i}_{t+1} = x^{i}_{t} - \eta(g^{i}_{t} + \rho(x^{i}_{t} - \tilde{x}_{t})) $ and $ \tilde{x}_{t+1} = \tilde{x}_{t} + \eta \sum_{i=1}^{p} \rho(x^{i}_{t} - \tilde{x}_{t}) $, where $ \rho $ controls the elasticity.
• The method supports both synchronous and asynchronous training modes, with a momentum-based variant for faster convergence.
• Communication is reduced by allowing local workers to run for multiple iterations before syncing with the master, controlled by a communication period $ \tau $.
• The algorithm is implemented in Torch with MVAPICH2 for GPU-CPU communication, and evaluated on CIFAR-10 and ImageNet using ResNet and AlexNet architectures.

Experimental results

Research questions

• RQ1 Can reducing communication frequency in distributed SGD training improve convergence speed and model performance?
• RQ2 Does introducing an elastic force between local and central parameters enhance exploration and lead to better generalization in deep learning?
• RQ3 How does the stability of EASGD compare to ADMM under the same round-robin communication scheme?
• RQ4 Can the asynchronous variant of EASGD maintain convergence and stability under practical communication constraints?
• RQ5 To what extent does EASGD outperform baseline methods like DOWNPOUR and MSGD in terms of training speed and test accuracy?

Key findings

• EASGD achieved faster convergence than DOWNPOUR and other baselines on CIFAR-10 and ImageNet, with up to 2.5× speedup in wall-clock time for equivalent test error levels.
• With a communication period $ \tau = 10 $, parameter communication time became negligible (1–11 seconds) compared to total training time (1254–1323 seconds), reducing communication overhead significantly.
• The asynchronous EAMSGD variant achieved the best performance, requiring less wall-clock time than all other methods to reach target test error levels on both CIFAR-10 and ImageNet.
• EASGD demonstrated stable convergence under the round-robin scheme, with stability guaranteed under a simple condition, unlike ADMM, which lacked such guarantees.
• The momentum-based EASGD variant improved convergence speed and generalization, particularly in asynchronous settings, with learning rate decay applied per local worker's clock.
• For $ p=16 $ workers, EASGD maintained high test accuracy even with reduced communication, showing robustness to communication delays and scaling efficiently.