[Paper Review] Straggler Mitigation in Distributed Optimization Through Data Encoding

Can Karakus, Yifan Sun+2 more|arXiv (Cornell University)|Nov 14, 2017

Sparse and Compressive Sensing Techniques20 references63 citations

TL;DR

The paper embeds redundancy in the data itself to mitigate stragglers in distributed optimization, enabling coding-oblivious gradient descent and L-BFGS to converge to an approximate solution using only a subset of nodes.

ABSTRACT

Slow running or straggler tasks can significantly reduce computation speed in distributed computation. Recently, coding-theory-inspired approaches have been applied to mitigate the effect of straggling, through embedding redundancy in certain linear computational steps of the optimization algorithm, thus completing the computation without waiting for the stragglers. In this paper, we propose an alternate approach where we embed the redundancy directly in the data itself, and allow the computation to proceed completely oblivious to encoding. We propose several encoding schemes, and demonstrate that popular batch algorithms, such as gradient descent and L-BFGS, applied in a coding-oblivious manner, deterministically achieve sample path linear convergence to an approximate solution of the original problem, using an arbitrarily varying subset of the nodes at each iteration. Moreover, this approximation can be controlled by the amount of redundancy and the number of nodes used in each iteration. We provide experimental results demonstrating the advantage of the approach over uncoded and data replication strategies.

Motivation & Objective

• Motivate straggler mitigation in large-scale distributed optimization.
• Propose data-encoding strategies that introduce redundancy directly in X and y.
• Enable coding-oblivious execution where nodes operate without knowledge of encoding.
• Prove convergence guarantees for gradient descent and L-BFGS under encoded data.
• Provide practical encoding schemes and empirical validation against uncoded and replication strategies.

Proposed method

• Encode data as tilde;X = S X and tilde;y = S y with redundancy factor beta; and solve f;min_{w} (1/(2b n)) ||S(Xw - y)||^2.
• Use only the fastest k out of m nodes to supply gradient components per iteration.
• Derive convergence guarantees showing deterministic linear convergence to a neighborhood of w* under a spectral condition on S.
• Present three coding matrix classes: equiangular tight frames (ETF), fast transforms, and random matrices.
• Adapt gradient descent and L-BFGS (with line search) to the encoded problem and analyze their behavior.
• Discuss privacy advantages from data encoding and potential generalizations to non-smooth or constrained problems.

Experimental results

Research questions

• RQ1 Can encoded data with redundancy yield convergence guarantees for batch methods when only a subset of worker updates is used per iteration?
• RQ2 What spectral properties must the encoding matrix S satisfy to ensure convergence to a neighborhood of the original optimum?
• RQ3 How do specific encoding schemes (ETF, fast transforms, random matrices) perform in terms of convergence and practicality for distributed optimization?
• RQ4 Can standard algorithms like gradient descent and L-BFGS be made coding-oblivious without modifying their core procedures?
• RQ5 What is the trade-off between redundancy level, number of responding nodes per iteration, and approximation quality of the solution?

Key findings

• Encoding the data with redundancy enables deterministic linear convergence to a neighborhood of w* using only a fraction of nodes per iteration.
• Three classes of coding matrices (ETF, fast transforms, random matrices) are proposed to satisfy the encoding properties needed for convergence.
• Experimental results show coded schemes outperform uncoded and replication strategies in ridge regression and matrix factorization tasks.
• Convergence guarantees can be tuned by redundancy level and the number of node updates waited for in each iteration.
• Privacy benefits arise since nodes operate on encoded data rather than raw data.
• The framework is extendable to more general objectives and constrained/non-smooth problems.