[Paper Review] GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training

Thomas Paine, Hailin Jin+3 more|arXiv (Cornell University)|Dec 21, 2013

Advanced Neural Network Applications14 references67 citations

TL;DR

This paper proposes GPU Asynchronous Stochastic Gradient Descent (GPU A-SGD), a hybrid approach combining model parallelism via multiple GPUs and data parallelism using asynchronous SGD to accelerate training of large convolutional neural networks. It achieves a 3.2x speedup on the ImageNet ILSVRC 2012 dataset with 8 GPUs while maintaining near-state-of-the-art performance, demonstrating that asynchronous updates with frequent parameter synchronization can significantly reduce training time without sacrificing accuracy.

ABSTRACT

The ability to train large-scale neural networks has resulted in state-of-the-art performance in many areas of computer vision. These results have largely come from computational break throughs of two forms: model parallelism, e.g. GPU accelerated training, which has seen quick adoption in computer vision circles, and data parallelism, e.g. A-SGD, whose large scale has been used mostly in industry. We report early experiments with a system that makes use of both model parallelism and data parallelism, we call GPU A-SGD. We show using GPU A-SGD it is possible to speed up training of large convolutional neural networks useful for computer vision. We believe GPU A-SGD will make it possible to train larger networks on larger training sets in a reasonable amount of time.

Motivation & Objective

• To accelerate large-scale neural network training by combining GPU-based model parallelism with asynchronous data parallelism.
• To evaluate the effectiveness of GPU A-SGD in reducing training time for large convolutional neural networks on the ImageNet ILSVRC 2012 benchmark.
• To investigate the impact of varying synchronization frequency ($n_{ ext{sync}}$) and number of GPU clients on training convergence and error rates.
• To explore the trade-offs between update frequency and training stability in asynchronous distributed training.
• To determine whether warm starts and adaptive optimization methods can improve performance in asynchronous GPU training.

Proposed method

• Extends the cuda-convnet framework to support multiple GPU clients communicating with a central parameter server using MPI for inter-process communication.
• Employs asynchronous stochastic gradient descent (A-SGD), where each GPU client independently computes gradients on mini-batches and asynchronously updates shared model parameters.
• Uses a synchronization interval $n_{ ext{sync}}$ to control how often clients fetch updated parameters from the server and push gradients back.
• Applies a sliding window smoothing technique (400 mini-batches) to training error curves to enable clearer comparison across different configurations.
• Implements both cold start and warm start training protocols to analyze initial learning dynamics and convergence behavior.
• Uses standard deep learning components: ReLU activations, dropout, data augmentation, and multi-crop testing for evaluation.

Experimental results

Research questions

• RQ1 Can GPU A-SGD achieve a significant speedup in training large convolutional neural networks compared to single-GPU training?
• RQ2 How does the number of GPU clients affect learning speed and final error rate, particularly in the early training phase?
• RQ3 What is the optimal synchronization frequency ($n_{ ext{sync}}$) for balancing update staleness and training efficiency?
• RQ4 Does a warm start mitigate the initial learning slowdown observed in high-client asynchronous training setups?
• RQ5 How do training dynamics differ between cold start and warm start configurations in terms of convergence and error progression?

Key findings

• GPU A-SGD achieved a 3.2x speedup on the ImageNet ILSVRC 2012 dataset using 8 GPUs, reducing training time from 10.6 days to 3.3 days while reaching a test error of 42.2%.
• With a warm start, training error dropped from 70% (single GPU) to 58% (8 GPUs) when $n_{ ext{sync}} = 100$, demonstrating significant performance gains with increased parallelism.
• Initial training with many GPU clients (e.g., 32) was significantly slower than with fewer clients, suggesting that gradient inconsistency early in training slows convergence.
• Later in training, models with more GPU clients exhibited steeper learning curves, indicating that gradient consistency improves over time and benefits from averaging.
• Higher synchronization frequency ($n_{ ext{sync}} = 100$) led to lower final error despite fewer total mini-batches processed, indicating that frequent updates can outweigh communication overhead.
• Jagged artifacts in error curves at high $n_{ ext{sync}}$ values (e.g., 900) suggest that stale gradients may cause instability and hinder convergence.