[Paper Review] Polynomially Coded Regression: Optimal Straggler Mitigation via Data Encoding

Songze Li, Seyed Mohammadreza Mousavi Kalan+3 more|arXiv (Cornell University)|May 24, 2018

Distributed Sensor Networks and Detection Algorithms20 references48 citations

TL;DR

PCR encodes data at workers to enable gradient computation via polynomial interpolation, dramatically reducing the straggler-robust recovery threshold for distributed least-squares regression.

ABSTRACT

We consider the problem of training a least-squares regression model on a large dataset using gradient descent. The computation is carried out on a distributed system consisting of a master node and multiple worker nodes. Such distributed systems are significantly slowed down due to the presence of slow-running machines (stragglers) as well as various communication bottlenecks. We propose "polynomially coded regression" (PCR) that substantially reduces the effect of stragglers and lessens the communication burden in such systems. The key idea of PCR is to encode the partial data stored at each worker, such that the computations at the workers can be viewed as evaluating a polynomial at distinct points. This allows the master to compute the final gradient by interpolating this polynomial. PCR significantly reduces the recovery threshold, defined as the number of workers the master has to wait for prior to computing the gradient. In particular, PCR requires a recovery threshold that scales inversely proportionally with the amount of computation/storage available at each worker. In comparison, state-of-the-art straggler-mitigation schemes require a much higher recovery threshold that only decreases linearly in the per worker computation/storage load. We prove that PCR's recovery threshold is near minimal and within a factor two of the best possible scheme. Our experiments over Amazon EC2 demonstrate that compared with state-of-the-art schemes, PCR improves the run-time by 1.50x ~ 2.36x with naturally occurring stragglers, and by as much as 2.58x ~ 4.29x with artificial stragglers.

Motivation & Objective

• Motivate and address slow stragglers and communication bottlenecks in distributed gradient descent for least-squares regression.
• Develop a data-encoding based scheme that reduces the gradient recovery threshold.
• Show theoretical optimality guarantees and practical gains via experiments on Amazon EC2.
• Extend the approach to kernelized/nonlinear regression problems via the kernel trick.
• Provide a complexity comparison with prior gradient coding schemes.

Proposed method

• Encode data across workers so each computes a polynomial evaluation through coded sub-matrices.
• Use a polynomial of degree 2⟨n/r⟩−2 per worker and interpolate from the fastest 2⟨n/r⟩−1 workers to recover the full gradient.
• Achieve recovery threshold KPCR(r)=2⌈n/r⌉−1, which is within a factor of 2 of the lower bound K*(r).
• Prove a lower bound showing any scheme needs at least ⌈n/r⌉ workers, hence the factor-2 gap is near-optimal.
• Compare PCR with gradient coding (GC) in terms of recovery threshold, decoding complexity, and communication.
• Demonstrate practical gains via Amazon EC2 experiments against GC, naive, and BCC schemes.

Experimental results

Research questions

• RQ1 What is the minimum recovery threshold achievable for coded distributed regression under a fixed per-worker storage/computation load r?
• RQ2 Can we design data-encoding based schemes that reduce the master’s waiting time for gradients beyond gradient-coding approaches?
• RQ3 How does PCR perform in practice compared to existing straggler-mitigation schemes on real distributed environments?
• RQ4 Can the PCR idea extend to nonlinear regression via kernels while maintaining straggler robustness?
• RQ5 What are the computational and communication trade-offs of PCR vs GC in GD iterations?

Key findings

• PCR achieves recovery threshold KPCR(r)=2⌈n/r⌉−1, improving over GC’s n−r+1 by ~r/2× for the same r.
• A near-optimal lower bound shows any scheme requires at least ⌈n/r⌉ workers, placing PCR within a factor of 2 of optimum.
• PCR’s decoding complexity scales as O(d(n/r) log^2(n/r) log log(n/r)) and does not grow with n for fixed r, unlike GC.
• Experiments on Amazon EC2 show PCR runs faster than GC by 1.50×–2.36× with natural stragglers and 2.58×–4.29× with artificial stragglers.
• PCR reduces per-iteration communication by requiring results from fewer workers (2⌈n/r⌉−1 instead of n−r+1).
• PCR can be extended to nonlinear regression via kernel methods by applying the kernel trick to the data matrix.