[Paper Review] Cascaded Coded Distributed Computing on Heterogeneous Networks

Nicholas Woolsey, Rong‐Rong Chen+1 more|arXiv (Cornell University)|Jan 23, 2019

Stochastic Gradient Optimization Techniques7 references4 citations

TL;DR

This paper proposes a novel combinatorial design for cascaded coded distributed computing (CDC) on heterogeneous networks, where nodes have varying storage and computing capabilities. By enabling multiple nodes to compute each Reduce function and leveraging coded data shuffling across strategically grouped nodes, the scheme achieves a computation-communication trade-off optimal within a constant factor and outperforms existing homogeneous CDC schemes in communication load while reducing encoding complexity.

ABSTRACT

Coded distributed computing (CDC) introduced by Li et al. in 2015 offers an efficient approach to trade computing power to reduce the communication load in general distributed computing frameworks such as MapReduce. For the more general cascaded CDC, Map computations are repeated at $r$ nodes to significantly reduce the communication load among nodes tasked with computing $Q$ Reduce functions $s$ times. While an achievable cascaded CDC scheme was proposed, it only operates on homogeneous networks, where the storage, computation load and communication load of each computing node is the same. In this paper, we address this limitation by proposing a novel combinatorial design which operates on heterogeneous networks where nodes have varying storage and computing capabilities. We provide an analytical characterization of the computation-communication trade-off and show that it is optimal within a constant factor and could outperform the state-of-the-art homogeneous schemes.

Motivation & Objective

• To address the limitation of existing coded distributed computing schemes that only operate on homogeneous networks with identical node capabilities.
• To design a scalable and efficient CDC framework for heterogeneous networks where nodes differ in storage, computation, and communication capacity.
• To enable cascaded CDC—where each Reduce function is computed at multiple nodes—while minimizing communication load through coded shuffling.
• To analytically characterize the computation-communication trade-off and demonstrate near-optimality within a constant factor.
• To reduce the complexity of encoding and decoding by minimizing random linear combinations over intermediate values.

Proposed method

• A novel combinatorial design is introduced to assign input files and intermediate values across nodes with heterogeneous storage and computing capabilities.
• The scheme organizes nodes into groups based on their capabilities, using sets $\mathcal{K}_i$ to represent nodes storing files for the $i$-th Map-Reduce round.
• In each of $s$ rounds of the Shuffle phase, intermediate values are transmitted in coded pairs using structured subsets $\mathcal{S}$, $\mathcal{S}'$, and $\mathcal{Y}$ to exploit coding gains.
• The communication load is derived by counting transmissions across all possible node groupings $\mathcal{A}$ of size $\gamma$, with $\gamma = 1$ to $s-1$, and the final round ($\gamma = s$) using symmetric coding across $2s$ nodes.
• The scheme normalizes the total transmitted bits by $QNT = \eta_1\eta_2X^2T$ to express the communication load as a function of system parameters.
• The design ensures that each intermediate value is computed at $r_c = s$ nodes, and the total computation load is normalized to $r_c = s$.

Experimental results

Research questions

• RQ1 Can a cascaded CDC scheme be designed for heterogeneous networks where nodes have unequal storage and computing capabilities?
• RQ2 What is the achievable computation-communication trade-off in such heterogeneous cascaded CDC systems?
• RQ3 How can coding gains be maximized while minimizing encoding and decoding complexity in heterogeneous settings?
• RQ4 Does the proposed scheme outperform existing homogeneous CDC schemes in terms of communication load?
• RQ5 Is the proposed trade-off optimal within a constant factor for heterogeneous networks?

Key findings

• The proposed scheme achieves a computation-communication trade-off that is optimal within a constant factor for certain system parameters, particularly when node capabilities are heterogeneous.
• The communication load is significantly reduced compared to state-of-the-art homogeneous CDC schemes, even when fixing other system parameters such as computation load and number of files.
• The scheme reduces the need for random linear combinations over intermediate values, thereby lowering encoding and decoding complexity in the Shuffle phase.
• The total number of bits transmitted in the Shuffle phase is derived as $\frac{\eta_1\eta_2}{2X} \sum_{\gamma=1}^{s-1} \left[ \sum_{\substack{\mathcal{A} \subset [s] \\ |\mathcal{A}|=\gamma}} \prod_{i \in \mathcal{A}} (x_i - 1) \right] + \frac{s\eta_1\eta_2TX}{2s-1} \prod_{i=1}^{s} (x_i - 1)$, which quantifies the communication load.
• The computation load is normalized to $r_c = s$, meaning each intermediate value is computed at $s$ nodes, enabling a multiplicative reduction in communication load by a factor of $s$.
• The analytical derivation confirms that the scheme maintains coding gains through structured grouping of nodes and intermediate value sets, ensuring efficient data shuffling.