[Paper Review] Distributed Deep Learning Using Synchronous Stochastic Gradient Descent

Dipankar Das, Sasikanth Avancha+6 more|arXiv (Cornell University)|Feb 22, 2016

Advanced Neural Network Applications11 references150 citations

TL;DR

The paper presents a multinode synchronous SGD framework (PCL-DNN) that scales vanilla SGD on CPUs without changing hyperparameters, achieving record throughput on CNNs and demonstrating strong scaling on both HPC clusters and AWS.

ABSTRACT

We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a detailed analysis of scaling, and identify optimal design points for different networks. We demonstrate scaling of CNNs on 100s of nodes, and present what we believe to be record training throughputs. A 512 minibatch VGG-A CNN training run is scaled 90X on 128 nodes. Also 256 minibatch VGG-A and OverFeat-FAST networks are scaled 53X and 42X respectively on a 64 node cluster. We also demonstrate the generality of our approach via best-in-class 6.5X scaling for a 7-layer DNN on 16 nodes. Thereafter we attempt to democratize deep-learning by training on an Ethernet based AWS cluster and show ~14X scaling on 16 nodes.

Motivation & Objective

• Motivate and enable large-scale training of deep networks using synchronous SGD on multinode CPU systems without altering hyperparameters or applying compression.
• Develop an analytically guided, hybrid data- and model-parallel approach to maximize scaling efficiency across networks and layers.
• Optimize single-node performance and network communications to achieve high overall time-to-train on Xeon-based clusters and clouds.
• Demonstrate practical scaling and throughput gains on CNNs (OverFeat, VGG-A) and DNNs (7-layer DNN, CD-DNN-HMM) across multiple platforms.

Proposed method

• Derive and solve balance equations for computation and memory bandwidth to obtain optimal cache-blocking and threading strategies.
• Propose and analyze data-parallel, model-parallel, and hybrid parallelism, including a quantitative comparison of data vs. model communication requirements.
• Develop a data-layout and vectorization strategy with cache/blocking optimizations to approach high single-node efficiency (up to ~90% for convs, ~70% for FC).
• Introduce deep-learning communication primitives (part-reduce and part-broadcast) implemented via MPI_Reduce_scatter and MPI_Allgather for scalable multinode training.
• Present the PCL-DNN software framework consisting of data handling, optimized compute library for CNNs/DNNs on x86 with AVX2, and MPI-based communications to enable large-scale execution.

Experimental results

Research questions

• RQ1 How can vanilla synchronous SGD be scaled across multinode CPU clusters without altering hyperparameters or using data compression?
• RQ2 What are the optimal data-, model-, and hybrid-parallel strategies for different network layers to maximize strong scaling on CPUs?
• RQ3 What single-node optimizations (cache blocking, vectorization, threading) are required to achieve high training throughput and time-to-train for CNNs on x86 CPUs?
• RQ4 How does the proposed framework perform on diverse platforms (Cori HPC cluster and AWS EC2) for CNNs like OverFeat and VGG-A, and for ASR-style DNNs?

Key findings

• A 512 minibatch VGG-A CNN training run scales 90x on 128 nodes with high throughput.
• A 256 minibatch VGG-A and OverFeat-FAST scale 53x and 42x respectively on 64 nodes.
• Best-in-class 6.5x scaling for a 7-layer DNN on 16 nodes demonstrates framework generality.
• On AWS, 14x scaling on 16 nodes is achieved for VGG-A with minimal hyperparameter changes.
• Single-node optimizations achieve about 90% efficiency for convolutional layers and 70% for fully-connected layers on Xeon processors.
• On Cori, VGG-A scales linearly up to 128 nodes with 70-82% efficiency for 256–512 minibatches, reducing epoch time to under 10 minutes on ImageNet-1K.