[Paper Review] ChainerMN: Scalable Distributed Deep Learning Framework

Takuya Akiba, Keisuke Fukuda+1 more|arXiv (Cornell University)|Oct 31, 2017

Advanced Neural Network ApplicationsComputer Science6 references58 citations

TL;DR

ChainerMN extends the Chainer framework to enable synchronous data-parallel distributed training across up to 128 GPUs, achieving high parallel efficiency. It uses NCCL for fast all-reduce and provides an easy porting path from single-node Chainer code.

ABSTRACT

One of the keys for deep learning to have made a breakthrough in various fields was to utilize high computing powers centering around GPUs. Enabling the use of further computing abilities by distributed processing is essential not only to make the deep learning bigger and faster but also to tackle unsolved challenges. We present the design, implementation, and evaluation of ChainerMN, the distributed deep learning framework we have developed. We demonstrate that ChainerMN can scale the learning process of the ResNet-50 model to the ImageNet dataset up to 128 GPUs with the parallel efficiency of 90%.

Motivation & Objective

• Motivate the need for scalable distributed learning to accelerate deep learning research and handle large datasets like ImageNet.
• Present a distributed extension (ChainerMN) for the Chainer framework that preserves Define-by-Run flexibility.
• Design an API and implementation that allows easy porting of existing Chainer programs to distributed execution.
• Demonstrate scalability and performance on a large GPU cluster using ResNet-50 on ImageNet.

Proposed method

• Adopt synchronous data-parallel training with Allreduce-based gradient averaging across workers.
• Integrate NCCL to optimize inter-GPU and inter-node communication for Allreduce.
• Provide a Define-by-Run friendly API that minimalistically extends existing Chainer programs (Communicator, Multi-node Optimizer, Scatter).
• Wrap Chainer optimizers with Multi-node Optimizer to transparently exchange gradients before parameter updates.
• Distribute dataset via Scatter to equal chunks across processes.
• Evaluate scalability on a 128-GPU cluster using ResNet-50 on ImageNet.

Experimental results

Research questions

• RQ1 Can ChainerMN achieve scalable synchronous data-parallel training for large models on large GPU clusters?
• RQ2 What is the achievable parallel efficiency when scaling up to 128 GPUs on a realistic workload (ResNet-50 on ImageNet)?
• RQ3 How does the ChainerMN API affect ease of porting existing Chainer code to distributed environments?
• RQ4 What performance benefits does using NCCL provide for Allreduce in this context?

Key findings

• ChainerMN scales to 128 GPUs with high parallel efficiency (90% relative to a single node and 79% relative to a single GPU).
• On 128 GPUs, ChainerMN achieves a speed-up of 101.32× over a single GPU for the ResNet-50/ImageNet experiment.
• The reported parallel efficiencies across configurations remain high (e.g., 88–92% range for smaller scales, 79.16% at 128 GPUs).
• The design preserves Chainer’s Define-by-Run flexibility while enabling distributed execution with minimal code changes.
• Using NCCL improves the efficiency of all-reduce communications, contributing to the observed scalability.