[Paper Review] ImageNet/ResNet-50 Training in 224 Seconds.

Hiroaki Mikami, Hisahiro Suganuma+3 more|arXiv (Cornell University)|Nov 13, 2018

Advanced Neural Network Applications5 references57 citations

TL;DR

This paper presents a method to train ImageNet/ResNet-50 in just 122 seconds using a large-scale GPU cluster by combining batch-size control and label smoothing to stabilize large mini-batch training, and 2D-Torus all-reduce to minimize gradient synchronization overhead. The approach achieves near-state-of-the-art accuracy with minimal degradation, demonstrating highly efficient distributed training at scale.

ABSTRACT

Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch-size control and label smoothing. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 122 seconds without significant accuracy loss on ABCI cluster.

Motivation & Objective

• To address the instability inherent in large mini-batch training for deep learning models like ResNet-50.
• To reduce the communication overhead of gradient synchronization in distributed training across massive GPU clusters.
• To enable end-to-end training of ImageNet/ResNet-50 in under two minutes while preserving model accuracy.
• To implement scalable and stable training using optimized collective communication and training techniques.

Proposed method

• Batch-size control is applied to stabilize training when using large mini-batches, mitigating optimization instability.
• Label smoothing is used to improve generalization and reduce overfitting in large mini-batch settings.
• 2D-Torus all-reduce organizes GPUs in a logical 2D grid to perform efficient, low-latency gradient synchronization across the cluster.
• The 2D-Torus all-reduce performs collective operations in multiple orientations to reduce communication bottlenecks.
• Neural Network Libraries (NNL) are used to implement and integrate the proposed techniques into the training pipeline.
• The system is deployed and evaluated on the ABCI GPU cluster to validate performance and accuracy.

Experimental results

Research questions

• RQ1 Can large mini-batch training be stabilized effectively without sacrificing model accuracy?
• RQ2 How can gradient synchronization overhead be minimized in large-scale distributed training?
• RQ3 What is the achievable training time for ImageNet/ResNet-50 using optimized communication and training techniques?
• RQ4 To what extent can training speed be improved while maintaining competitive accuracy?

Key findings

• ImageNet/ResNet-50 was successfully trained in 122 seconds on the ABCI GPU cluster using the proposed method.
• The training achieved near-state-of-the-art accuracy with minimal degradation compared to standard training.
• The combination of batch-size control and label smoothing effectively stabilized large mini-batch training.
• The 2D-Torus all-reduce communication pattern significantly reduced synchronization overhead.