[Paper Review] Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks

Zhihao Jia, Sina Lin+2 more|arXiv (Cornell University)|Feb 14, 2018

Advanced Neural Network Applications21 references51 citations

TL;DR

Introduces layer-wise parallelism that allows each CNN layer to use its own parallelization strategy, optimizing per-layer execution via a graph-based cost model to improve throughput and reduce communication without sacrificing accuracy.

ABSTRACT

The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our evaluation shows that layer-wise parallelism outperforms state-of-the-art approaches by increasing training throughput, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining original network accuracy.

Motivation & Objective

• Motivate the need for flexible parallelization across CNN layers due to heterogeneous layer characteristics.
• Propose layer-wise parallelism to allow per-layer configuration and preserve network accuracy.
• Develop a cost model and graph-search algorithm to find globally optimal per-layer parallelization strategies.
• Demonstrate throughput and communication improvements over state-of-the-art methods on standard CNNs.

Proposed method

• Define a parallelization configuration for each layer as a product of per-dimension degrees of parallelism.
• Model computation and communication costs with t_c, t_x, and t_s to estimate per-layer and inter-layer timings.
• Convert the optimization into a graph-search problem over a device graph and a computation graph using dynamic programming.
• Apply node and edge elimination reductions to simplify the computation graph while preserving optimal strategies.
• Provide an algorithm (Algorithm 1) to compute a globally optimal per-layer strategy under the cost model.

Experimental results

Research questions

• RQ1 Can layer-wise parallelism outperform single-strategy approaches (data or model parallelism) for CNN training?
• RQ2 How can a cost model and graph-based search identify per-layer parallelization configurations that minimize per-iteration time?
• RQ3 What is the impact of per-layer parallelization on throughput and inter-device communication across standard CNNs?
• RQ4 How scalable is the proposed approach to large GPU clusters and different CNN architectures?
• RQ5 Do the learned strategies preserve the original network accuracy?

Key findings

• Layer-wise parallelism yields 1.4–2.2× faster training throughput than state-of-the-art strategies on 16 GPUs across AlexNet, VGG-16, and Inception-v3.
• Layer-wise parallelism reduces communication costs by 1.3–23.0× compared to data and model parallelism.
• Layer-wise parallelism achieves better scalability, e.g., 15.5× speedup for Inception-v3 from 1 to 16 GPUs versus up to 11.2× for other strategies.
• The cost model estimates per-step execution time within ~10% of measured times across tested configurations.
• The approach trains the same network as data/model parallelism, preserving original network accuracy while improving runtime performance.