[Paper Review] Efficient Training of Convolutional Neural Nets on Large Distributed Systems

Sameer Kumar, Dheeraj Sreedhar+3 more|arXiv (Cornell University)|Nov 2, 2017

Advanced Neural Network Applications18 references3 citations

TL;DR

This paper proposes three key optimizations for efficient distributed training of convolutional neural networks: in-memory data loading to eliminate I/O bottlenecks, a multi-color MPI Allreduce algorithm to reduce communication overhead, and thread-level optimizations in Torch's data parallel table. These techniques enabled training ResNet-50 on ImageNet-1k with 256 GPUs in 48 minutes—18% faster than the prior state of the art—while maintaining peak accuracy.

ABSTRACT

Deep Neural Networks (DNNs) have achieved im- pressive accuracy in many application domains including im- age classification. Training of DNNs is an extremely compute- intensive process and is solved using variants of the stochastic gradient descent (SGD) algorithm. A lot of recent research has focussed on improving the performance of DNN training. In this paper, we present optimization techniques to improve the performance of the data parallel synchronous SGD algorithm using the Torch framework: (i) we maintain data in-memory to avoid file I/O overheads, (ii) we present a multi-color based MPI Allreduce algorithm to minimize communication overheads, and (iii) we propose optimizations to the Torch data parallel table framework that handles multi-threading. We evaluate the performance of our optimizations on a Power 8 Minsky cluster with 32 nodes and 128 NVidia Pascal P100 GPUs. With our optimizations, we are able to train 90 epochs of the ResNet-50 model on the Imagenet-1k dataset using 256 GPUs in just 48 minutes. This significantly improves on the previously best known performance of training 90 epochs of the ResNet-50 model on the same dataset using 256 GPUs in 65 minutes. To the best of our knowledge, this is the best known training performance demonstrated for the Imagenet- 1k dataset.

Motivation & Objective

• Address the performance bottleneck caused by slow file I/O during mini-batch data loading in distributed DNN training.
• Reduce communication overhead in synchronous data-parallel SGD by optimizing the MPI Allreduce collective operation.
• Improve multi-threading efficiency in the Torch framework’s data parallel table to better utilize multi-GPU systems.
• Achieve faster training times for large models like ResNet-50 and GoogLeNet on large-scale GPU clusters without sacrificing model accuracy.
• Demonstrate the best-known training performance for ImageNet-1k using 256 GPUs, setting a new performance standard in distributed DNN training.

Proposed method

• Implement an in-memory data distribution strategy that preloads the entire ImageNet-1k dataset into distributed system memory to eliminate repeated disk I/O during training iterations.
• Design a multi-color-based MPI Allreduce algorithm that reduces communication latency by overlapping computation and communication phases across GPU nodes.
• Optimize the Torch framework’s data parallel table module to improve thread-level load balancing and reduce contention in multi-GPU, multi-node training environments.
• Use the Torch deep learning framework with its Lua scripting and C-based FFI for efficient low-level computation, enabling high-performance distributed training.
• Integrate the optimizations into a distributed Torch application using OpenMPI and evaluate performance on a POWER8 Minsky cluster with 128 P100 GPUs.
• Measure training time per epoch and validation accuracy across varying node counts (8, 16, 32) to assess scalability and correctness.

Experimental results

Research questions

• RQ1 How can file I/O overhead be minimized in distributed DNN training when using large-scale datasets like ImageNet-1k?
• RQ2 What communication optimization techniques can significantly reduce latency in the MPI Allreduce step of synchronous data-parallel SGD?
• RQ3 To what extent can thread-level inefficiencies in the Torch data parallel table framework limit training performance on multi-GPU systems?
• RQ4 Can the combination of I/O, communication, and threading optimizations achieve a significant speedup in training large models like ResNet-50 without accuracy degradation?
• RQ5 How does the proposed system compare to state-of-the-art implementations in terms of training time and model accuracy on the same hardware and dataset?

Key findings

• The in-memory data loading strategy reduced file I/O overhead, significantly improving data throughput and reducing per-epoch training time.
• The multi-color MPI Allreduce algorithm decreased communication latency, contributing to a 15–18% reduction in per-epoch time across both GoogLeNet and ResNet-50 models.
• The optimized data parallel table in Torch improved thread utilization and reduced synchronization bottlenecks, especially on larger node counts.
• With all optimizations applied, ResNet-50 training on ImageNet-1k with 256 GPUs completed 90 epochs in 48 minutes—18% faster than the prior best-known time of 65 minutes.
• The peak validation accuracy of 75.4% was achieved with the optimized setup, comparable to the 76.2% reported in the prior state-of-the-art, confirming no accuracy degradation.
• The optimizations achieved a 110–130% speedup for ResNet-50 and 58–72% for GoogLeNetBN across different node counts, demonstrating strong scalability and performance gains.