[Paper Review] Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters

Hao Zhang, Zeyu Zheng+8 more|arXiv (Cornell University)|Jun 10, 2017

Advanced Neural Network Applications29 references197 citations

TL;DR

Poseidon introduces a layered, wait-free, hybrid communication architecture for data-parallel distributed deep learning on GPUs, achieving near-linear scaling by overlapping computation and communication and selecting optimal per-layer communication methods.

ABSTRACT

Deep learning models can take weeks to train on a single GPU-equipped machine, necessitating scaling out DL training to a GPU-cluster. However, current distributed DL implementations can scale poorly due to substantial parameter synchronization over the network, because the high throughput of GPUs allows more data batches to be processed per unit time than CPUs, leading to more frequent network synchronization. We present Poseidon, an efficient communication architecture for distributed DL on GPUs. Poseidon exploits the layered model structures in DL programs to overlap communication and computation, reducing bursty network communication. Moreover, Poseidon uses a hybrid communication scheme that optimizes the number of bytes required to synchronize each layer, according to layer properties and the number of machines. We show that Poseidon is applicable to different DL frameworks by plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even with limited bandwidth (10GbE) and the challenging VGG19-22K network for image classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up with 32 single-GPU machines on Inception-V3, a 50% improvement over the open-source TensorFlow (20x speed-up).

Motivation & Objective

• Motivate the need for scalable distributed DL on GPU clusters due to bursty and large-volume parameter synchronization.
• Propose Poseidon to overlap computation and communication using the layer-by-layer structure of DL models.
• Introduce a hybrid communication scheme to pick the cheapest synchronization method per layer.
• Demonstrate cross-framework applicability by integrating Poseidon with Caffe and TensorFlow.

Proposed method

• Decompose DL training into layer-wise computation and synchronization steps to enable overlap between forward/backward passes and communication.
• Introduce Wait-free Backpropagation (WFBP) to overlap gradient synchronization with lower-layer computation by scheduling independent operations concurrently.
• Propose Hybrid Communication (HybComm) to select the optimal synchronization method per layer (PS, SFB, Adam-like strategies) based on layer properties and cluster configuration.
• Implement Poseidon as a three-component system (coordinator, KV store, client library) exposing APIs to manage communication schedules and transfers.
• Demonstrate integration into existing frameworks (Caffe and TensorFlow) with minimal code changes and near-linear scalability up to 32 GPUs.

Experimental results

Research questions

• RQ1 How can DL training be reorganized to hide communication costs and reduce network burstiness on GPU clusters?
• RQ2 Can a per-layer hybrid communication strategy improve throughput compared to standard PS or SFB schemes at varying bandwidths and model sizes?
• RQ3 To what extent can Poseidon achieve near-linear throughput scaling across multiple DL frameworks and large-scale models?
• RQ4 What is the impact of Poseidon on convergence speed and overall training efficiency across representative CNNs and datasets?

Key findings

• Poseidon achieves near-linear throughput scaling on up to 32 Titan X GPUs across multiple models and frameworks.
• On 32 nodes, TensorFlow with Poseidon attains 31.5x speed-up for Inception-V3 and outperforms the original TensorFlow by 50% in speed-up.
• On 16 machines with limited 10GbE bandwidth, Poseidon maintains better scaling for large models (e.g., VGG19-22K) than PS-based parallelization.
• Poseidon reduces network communication bottlenecks by automatically specializing the best per-layer communication method, improving bandwidth utilization.
• Compared with SF strategies like Adam or 1-bit quantization in CNTK, Poseidon offers either higher algorithm throughput or stronger statistical performance stability.