[Paper Review] A New Combinatorial Design of Coded Distributed Computing

Nicholas Woolsey, Rong‐Rong Chen+1 more|arXiv (Cornell University)|Feb 12, 2018

Stochastic Gradient Optimization Techniques4 references3 citations

TL;DR

This paper proposes a novel coded distributed computing scheme based on hypercube combinatorial design that drastically reduces the number of required input files and Map functions compared to prior work. By leveraging structured file placement and coded multicasting, the scheme achieves a multiplicative communication load reduction while maintaining near-optimal performance and asymptotic optimality under certain conditions.

ABSTRACT

Coded distributed computing introduced by Li et al. in 2015 is an efficient approach to trade computing power to reduce the communication load in general distributed computing frameworks such as MapReduce. In particular, Li et al. show that increasing the computation load in the Map phase by a factor of $r$ can create coded multicasting opportunities to reduce the communication load in the Reduce phase by the same factor. However, there are two major limitations in practice. First, it requires an exponentially large number of input files (data batches) when the number of computing nodes gets large. Second, it forces every $s$ computing nodes to compute one Map function, which leads to a large number of Map functions required to achieve the promised gain. In this paper, we make an attempt to overcome these two limitations by proposing a novel coded distributed computing approach based on a combinatorial design. We demonstrate that when the number of computing nodes becomes large, 1) the proposed approach requires an exponentially less number of input files; 2) the required number of Map functions is also reduced exponentially. Meanwhile, the resulting computation-communication trade-off maintains the multiplicative gain compared to conventional uncoded unicast and achieves the information theoretic lower bound asymmetrically for some system parameters.

Motivation & Objective

• To address the two major limitations of existing coded distributed computing: exponential growth in required input files and Map functions as the number of nodes increases.
• To reduce the number of input files and Map functions needed to achieve the same computation-communication trade-off gain.
• To maintain a multiplicative reduction in communication load compared to uncoded unicast.
• To achieve information-theoretic optimality asymptotically for specific parameter regimes such as s=1 and s=2.
• To design a scalable scheme that requires significantly fewer system resources while preserving performance.

Proposed method

• The scheme uses a hypercube-based combinatorial design to structure file placement across nodes, ensuring that intermediate values needed by multiple nodes are locally available at multiple locations.
• Each node computes only the necessary intermediate values (Map functions) for its assigned output functions, avoiding redundant computation.
• The communication load is minimized through coded multicasting, where each node transmits random linear combinations of its locally computed packets to satisfy requests from other nodes.
• The scheme is formalized using parameters x, d, and η₁, η₂ to define node count K = xd, number of files N = η₁x^d, and number of functions Q = η₂xd.
• The computation load r_hc and communication load L_hc are derived using combinatorial expressions involving binomial coefficients and powers of x.
• The design ensures that each intermediate value is requested by s nodes and can be computed by r nodes, enabling efficient coded transmission.

Experimental results

Research questions

• RQ1 Can a coded distributed computing scheme be designed to reduce the exponential growth in required input files and Map functions as the number of nodes increases?
• RQ2 Does the proposed scheme maintain a multiplicative gain in communication load reduction compared to uncoded unicast?
• RQ3 Can the scheme achieve information-theoretic optimality for certain system parameters such as s=1 or s=2?
• RQ4 How does the required number of files and functions scale with system size in the proposed scheme versus prior work?
• RQ5 What is the trade-off between computation load and communication load in the proposed hypercube-based design?

Key findings

• The proposed scheme reduces the required number of input files N from Κ choose r to (K/r)^r, achieving an exponential reduction in file requirements for large r.
• The number of Map functions required is reduced from Κ choose s to (K/s)^s, resulting in an exponential decrease as s increases.
• The communication load L_hc achieves a multiplicative gain over uncoded unicast, with L_hc = 20/27 when K=9, r=14/9, and s=3, compared to L*=8/15 in the optimal scheme.
• For s=1, the scheme achieves information-theoretic optimality as r→∞, demonstrating asymptotic optimality.
• For s=2, the scheme achieves information-theoretic optimality as K→∞, confirming asymptotic optimality under this regime.
• The scheme maintains near-optimal performance with L_hc close to the theoretical lower bound L* from [1], especially when s=r is small.