[Paper Review] Asynchronous Stochastic Gradient Descent with Delay Compensation for Distributed Deep Learning.

Shuxin Zheng, Qi Meng+5 more|arXiv (Cornell University)|Sep 27, 2016

Advanced Neural Network Applications18 references27 citations

TL;DR

This paper proposes Delay Compensated ASGD (DC-ASGD), a novel asynchronous stochastic gradient descent method that uses Taylor expansion and Hessian approximation to correct gradient delays in distributed deep learning. The method significantly improves convergence and performance, outperforming both synchronous and standard asynchronous SGD and approaching sequential SGD accuracy on CIFAR-10 and ImageNet.

ABSTRACT

With the fast development of deep learning, it has become common to learn big neural networks using massive training data. Asynchronous Stochastic Gradient Descent (ASGD) is widely adopted to fulfill this task for its efficiency, which is, however, known to suffer from the problem of delayed gradients. That is, when a local worker adds its gradient to the global model, the global model may have been updated by other workers and this gradient becomes delayed. We propose a novel technology to compensate this delay, so as to make the optimization behavior of ASGD closer to that of sequential SGD. This is achieved by leveraging Taylor expansion of the gradient function and efficient approximation to the Hessian matrix of the loss function. We call the new algorithm Delay Compensated ASGD (DC-ASGD). We evaluated the proposed algorithm on CIFAR-10 and ImageNet datasets, and the experimental results demonstrate that DC-ASGD outperforms both synchronous SGD and asynchronous SGD, and nearly approaches the performance of sequential SGD.

Motivation & Objective

• To address the performance degradation in asynchronous stochastic gradient descent (ASGD) caused by delayed gradients in distributed training.
• To improve optimization stability and convergence speed in large-scale deep learning by compensating for gradient delays.
• To bridge the performance gap between asynchronous SGD and sequential SGD in distributed training scenarios.
• To develop an efficient, scalable method that maintains high training accuracy while preserving the speed of asynchronous updates.

Proposed method

• The method employs Taylor expansion of the gradient function to model the change in gradient over time due to model updates.
• It approximates the Hessian matrix of the loss function to estimate the curvature and correct for delayed gradients.
• Gradient delay compensation is applied by adjusting the gradient update using the estimated Hessian and time delay information.
• The compensation mechanism is integrated into the ASGD framework to maintain computational efficiency in distributed systems.
• The approach is designed to be lightweight and scalable, suitable for large-scale deep learning workloads.

Experimental results

Research questions

• RQ1 Can gradient delay compensation improve the convergence and accuracy of asynchronous SGD in distributed deep learning?
• RQ2 How does the proposed delay compensation method compare to synchronous and standard asynchronous SGD in terms of training performance?
• RQ3 To what extent can the method approach the performance of sequential SGD in distributed settings?
• RQ4 How effective is the Hessian approximation in modeling gradient delay effects across different datasets?

Key findings

• DC-ASGD outperforms both synchronous SGD and standard asynchronous SGD on the CIFAR-10 and ImageNet datasets.
• The method achieves test accuracy close to that of sequential SGD, significantly reducing the performance gap seen in standard ASGD.
• Experimental results show improved convergence stability and faster training progress due to effective delay compensation.
• The use of Hessian approximation enables accurate gradient correction without incurring prohibitive computational overhead.