[Paper Review] Horovod: fast and easy distributed deep learning in TensorFlow

Alexander Sergeev, Mike Del Balso|arXiv (Cornell University)|Feb 15, 2018

Advanced Neural Network Applications5 references522 citations

TL;DR

Horovod introduces a ring-allreduce based distributed TensorFlow framework that dramatically reduces code changes and improves scaling, enabling near-linear speedups across multiple GPUs. It provides a standalone Python package, NCCL-backed communication, and debugging/ profiling tools.

ABSTRACT

Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead. Second, the user must modify his or her training code to take advantage of inter-GPU communication. Depending on the training library's API, the modification required may be either significant or minimal. Existing methods for enabling multi-GPU training under the TensorFlow library entail non-negligible communication overhead and require users to heavily modify their model-building code, leading many researchers to avoid the whole mess and stick with slower single-GPU training. In this paper we introduce Horovod, an open source library that improves on both obstructions to scaling: it employs efficient inter-GPU communication via ring reduction and requires only a few lines of modification to user code, enabling faster, easier distributed training in TensorFlow. Horovod is available under the Apache 2.0 license at https://github.com/uber/horovod

Motivation & Objective

• Motivate the need for scalable distributed TensorFlow training at Uber and identify two main barriers: inter-GPU communication overhead and user code complexity.
• Propose a ring-allreduce based approach to address scalability and simplicity.
• Describe Horovod's architecture, integration with TensorFlow/Keras, and API design to minimize user edits.
• Demonstrate practical tooling (Horovod Timeline) and optimizations (Tensor Fusion) to enhance usability and performance.

Proposed method

• Adopted ring-allreduce from Baidu's draft and replaced with NVIDIA NCCL for optimized cross-GPU and cross-machine communication.
• Implemented Horovod as a standalone Python package to decouple from specific TensorFlow releases.
• Extended support to models fitting in a single server (potentially multi-GPU).
• Introduced a broadcast initialization hook to ensure consistent startup across workers.
• Provided a minimal API surface enabling users to wrap optimizers with hvd.DistributedOptimizer and broadcast variables from rank 0.
• Incorporated Horovod Timeline for cross-node profiling and debugging.
• Developed Tensor Fusion to fuse small tensors into larger buffers before allreduce to improve throughput on TCP networks.

Experimental results

Research questions

• RQ1 Can ring-allreduce-based communication provide near-linear scaling for TensorFlow training across multiple GPUs and machines?
• RQ2 How much code modification is required to convert single-GPU TensorFlow programs to distributed Horovod programs?
• RQ3 What practical tooling and optimizations (like Tensor Fusion and Timeline) improve usability and performance in real-world workflows?
• RQ4 What are the performance characteristics of Horovod on TCP vs RDMA networks and for models with varying parameter counts?
• RQ5 How does Horovod compare to standard distributed TensorFlow in terms of efficiency and resource utilization?

Key findings

• Horovod achieves substantial scaling improvements over standard distributed TensorFlow, with up to 88% efficiency reported in benchmarks.
• Training speed can be nearly doubled relative to standard distributed TensorFlow when using Horovod on multiple GPUs.
• RDMA networking provides modest gains on some models (3-4% extra) and can push scaling efficiency above 90% for certain architectures.
• Tensor Fusion yields up to 65% improvement for models with many small tensor operations, by reducing communication overhead.
• Horovod reduces setup and integration effort to a few lines of code changes, enabling easier adoption across teams.
• Horovod Timeline provides high-level, browser-accessible profiling to aid debugging and performance analysis.