[Paper Review] Natural Compression for Distributed Deep Learning

Samuel Horváth, Chen-Yu Ho+4 more|arXiv (Cornell University)|May 27, 2019

Stochastic Gradient Optimization Techniques44 references69 citations

TL;DR

The paper introduces natural compression C_nat, which rounds each update entry to the nearest power of two via randomized rounding, achieving substantial communication savings with negligible impact on convergence, and extends to natural dithering for more aggressive compression with exponential improvements over standard dithering.

ABSTRACT

Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among machines becomes a significant performance bottleneck and various lossy update compression techniques have been proposed to alleviate this problem. In this work, we introduce a new, simple yet theoretically and practically effective compression technique: natural compression (NC). Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa. We show that compared to no compression, NC increases the second moment of the compressed vector by not more than the tiny factor $\frac{9}{8}$, which means that the effect of NC on the convergence speed of popular training algorithms, such as distributed SGD, is negligible. However, the communications savings enabled by NC are substantial, leading to $3$-$4 imes$ improvement in overall theoretical running time. For applications requiring more aggressive compression, we generalize NC to natural dithering, which we prove is exponentially better than the common random dithering technique. Our compression operators can be used on their own or in combination with existing operators for a more aggressive combined effect and offer new state-of-the-art both in theory and practice.

Motivation & Objective

• Motivate and address communication bottlenecks in data-parallel distributed deep learning.
• Propose a simple, unbiased compression operator with provably low variance.
• Show that compression yields substantial communication savings with negligible convergence slowdown.
• Introduce natural dithering for more aggressive compression and analyze its theoretical benefits.
• Demonstrate practical performance improvements and compatibility with existing compression methods.

Proposed method

• Define and implement natural compression C_nat that maps each real-valued update entry to a random power-of-two via unbiased rounding.
• Prove that C_nat belongs to the unbiased with bounded second moment class B(1/8), ensuring negligible impact on convergence (Theorem 2.3).
• Show how natural compression reduces communication by encoding only sign and exponent bits in IEEE 754 formats (3.56x fewer bits for float32, 5.82x for float64).
• Introduce natural dithering D_nat^{p,s} as an exponential improvement over standard dithering, and prove its variance and compression properties (Theorem 3.2, 3.3).
• Develop a bidirectional compression framework for distributed SGD (Algorithm 1) with master and workers using compression in B(ω) to achieve speedups (Theorem 4.1).
• Demonstrate compatibility with existing compression operators via composition rules (Theorem 2.5).
• Provide a proof-of-concept system and experiments validating training time reductions and scalability (ResNet110, AlexNet on CIFAR-10; ImageNet results).

Experimental results

Research questions

• RQ1 How much does natural compression increase the second moment of the update vector, and does it meaningfully affect convergence?
• RQ2 Can bidirectional compression with C_nat and natural dithering provide practical speedups in distributed SGD while maintaining accuracy?
• RQ3 What are the theoretical guarantees and practical benefits when combining natural compression with existing compression techniques?
• RQ4 How does natural dithering compare to standard dithering in terms of variance and efficiency under fixed communication budgets?

Key findings

• C_nat increases the second moment by at most a factor of 9/8, producing negligible impact on convergence for SGD-based methods.
• C_nat provides 3.2×–3.6× reduction in per-iteration communication with two-sided (bidirectional) compression.
• Natural dithering D_nat^{p,s} is exponentially better than standard dithering at the same variance level.
• Combined with sparsification or other operators, natural compression yields larger overall speedups than standard approaches (as shown in Table 1 discussion).
• Empirical results show significant training time reductions (e.g., ~26% for ResNet110 on CIFAR-10 and ~66% for AlexNet) without loss in final accuracy, and successful scalability in larger models like ImageNet.
• The proposed operators are compatible with SwitchML-style in-network aggregation and support a broad family of compression operators within B(ω).