[Paper Review] Communication-Computation Efficient Gradient Coding

Min Ye, Emmanuel Abbé|arXiv (Cornell University)|Feb 9, 2018

Stochastic Gradient Optimization Techniques73 citations

TL;DR

The paper introduces a three-way tradeoff between computation load, straggler tolerance, and communication cost for gradient summations, and presents a recursive polynomial coding scheme achieving the optimal tradeoff with exact recovery under given conditions.

ABSTRACT

This paper develops coding techniques to reduce the running time of distributed learning tasks. It characterizes the fundamental tradeoff to compute gradients (and more generally vector summations) in terms of three parameters: computation load, straggler tolerance and communication cost. It further gives an explicit coding scheme that achieves the optimal tradeoff based on recursive polynomial constructions, coding both across data subsets and vector components. As a result, the proposed scheme allows to minimize the running time for gradient computations. Implementations are made on Amazon EC2 clusters using Python with mpi4py package. Results show that the proposed scheme maintains the same generalization error while reducing the running time by $32\%$ compared to uncoded schemes and $23\%$ compared to prior coded schemes focusing only on stragglers (Tandon et al., ICML 2017).

Motivation & Objective

• Motivate the need to accelerate distributed gradient computations in the presence of stragglers and high communication costs.
• Formulate a three-parameter tradeoff among computation load, straggler tolerance, and communication reduction.
• Derive conditions for achievable gradient coding schemes that enable exact gradient recovery.
• Propose a recursive polynomial construction leveraging Vandermonde matrices to achieve the tradeoff.
• Demonstrate practical viability with experiments on Amazon EC2 showing runtime reductions.

Proposed method

• Define a three-parameter achievable region (d, s, m) where d/k ≥ (s+m)/n and aim for linear combining by workers.
• Construct a coding scheme using recursive polynomials to generate each worker’s output from its assigned partial gradients.
• Partition gradient coordinates into m groups to reduce transmitted dimension.
• Use a (n-s)×n Vandermonde-like matrix V and an (mn)×(n-s) matrix B with designed properties to enable exact recovery of the sum gradient from any n−s workers.
• Represent each worker’s transmission as f_i(g_i, g_{i⊕1}, ..., g_{i⊕(d-1)}) with f_i linear, ensuring recoverability of g_1+...+g_n from any subset of size n−s.
• Provide an efficient implementation strategy for computing B and the transmitted vectors, including a specified choice of theta parameters for numerical stability.

Experimental results

Research questions

• RQ1 What is the fundamental tradeoff between computation load, straggler tolerance, and communication cost in distributed gradient coding?
• RQ2 Can we achieve optimal recovery of the full gradient from a subset of workers with a linear coding scheme?
• RQ3 How can recursive polynomial constructions enable dimensionality reduction of transmitted gradients while preserving recoverability?
• RQ4 What are the numerical stability considerations for Vandermonde-based constructions and how do they impact achievable regions?
• RQ5 Do the proposed schemes yield practical runtime improvements in real distributed systems without sacrificing generalization performance?

Key findings

• The paper establishes a three-dimensional tradeoff: d/k ≥ (s+m)/n (and equivalent d ≥ s+m when n=k).
• An explicit coding scheme based on recursive polynomials achieves the tradeoff with linear f_i functions.
• Partitioning gradient coordinates into m groups reduces per-worker communication dimension to l/(dn−s) (under suitable divisibility assumptions).
• Vandermonde-based construction and a recursive polynomial design enable exact gradient recovery from any n−s workers, subject to numerical stability constraints.
• The approach reduces running time in experiments by 32% versus uncoded schemes and 23% versus prior coded schemes, while maintaining the same generalization error on a real dataset (Amazon Employee Access/Kaggle).