[Paper Review] Heterogeneous Coded Distributed Computing: Joint Design of File Allocation and Function Assignment

Fan Xu, Meixia Tao|arXiv (Cornell University)|Aug 19, 2019

Stochastic Gradient Optimization Techniques14 references4 citations

TL;DR

This paper proposes a joint file allocation and function assignment scheme for heterogeneous coded distributed computing systems, where nodes have varying computation capabilities. By partitioning files into compulsory and optional batches and leveraging coded multicasting in the shuffle phase, the scheme achieves a communication load within a constant multiplicative gap (up to 20e) of the optimal in an equivalent homogeneous system, outperforming prior works in certain scenarios.

ABSTRACT

This paper studies the computation-communication tradeoff in a heterogeneous MapReduce computing system where each distributed node is equipped with different computation capability. We first obtain an achievable communication load for any given computation load and any given function assignment at each node. The proposed file allocation strategy has two steps: first, the input files are partitioned into disjoint batches, each with possibly different size and computed by a distinct node; then, each node computes additional files from its non-computed files according to its redundant computation capability. In the Shuffle phase, coded multicasting opportunities are exploited thanks to the repetitive file allocation among different nodes. Based on this scheme, we further propose the computation-aware and the shuffle-aware function assignments. We prove that, by using proper function assignments, our achievable communication load for any given computation load is within a constant multiplicative gap to the optimum in an equivalent homogeneous system with the same average computation load. Numerical results show that our scheme with shuffle-aware function assignment achieves better computation-communication tradeoff than existing works in some cases.

Motivation & Objective

• Address the computation-communication tradeoff in heterogeneous MapReduce systems where nodes have unequal computation and storage capabilities.
• Design a file allocation strategy that exploits redundant computation capabilities to create coded multicasting opportunities in the shuffle phase.
• Propose computation-aware and shuffle-aware function assignment schemes to optimize communication load under heterogeneous node capabilities.
• Prove that the proposed scheme achieves a communication load within a constant multiplicative gap (up to 20e) of the optimal in an equivalent homogeneous system.
• Demonstrate through numerical results that the scheme outperforms existing approaches in specific heterogeneous configurations.

Proposed method

• Partition N input files into K disjoint batches: compulsory files (one per node) and optional files (computed based on redundant capacity).
• Assign Map functions to nodes based on their computation load m_k, with each node computing its compulsory files and additional optional files from other nodes.
• Use coded multicasting in the shuffle phase by exploiting file repetitions across nodes due to optional file computation.
• Propose two function assignment strategies: computation-aware (assigning more functions to higher-capacity nodes) and shuffle-aware (optimizing for coded multicasting gains).
• Derive a closed-form expression for achievable communication load under any given computation load and function assignment.
• Use asymptotic analysis and case-based bounding techniques to prove the multiplicative gap to the optimal load in an equivalent homogeneous system.

Experimental results

Research questions

• RQ1 Can a joint design of file allocation and function assignment achieve near-optimal communication load in a heterogeneous coded distributed computing system?
• RQ2 How does the proposed file partitioning strategy—compulsory and optional files—improve coded multicasting opportunities in the shuffle phase?
• RQ3 What is the maximum multiplicative gap between the proposed scheme's communication load and the optimal load in an equivalent homogeneous system?
• RQ4 How do computation-aware and shuffle-aware function assignments affect the communication-communication tradeoff in heterogeneous systems?
• RQ5 Under what conditions does the proposed scheme outperform existing schemes in terms of communication load?

Key findings

• The proposed scheme achieves a communication load within a constant multiplicative gap (up to 20e) of the optimal in an equivalent homogeneous system with the same average computation load.
• The shuffle-aware function assignment strategy leads to better computation-communication tradeoff than existing works in certain heterogeneous configurations.
• The communication load is derived in closed-form for any given computation load and function assignment, enabling systematic optimization.
• Numerical results confirm that the scheme outperforms prior works in terms of communication load reduction, especially when node capabilities are highly heterogeneous.
• The analysis proves that even with non-uniform node capabilities, the scheme maintains strong performance guarantees via bounded multiplicative gaps.
• The method of partitioning files into compulsory and optional batches effectively leverages redundant computation to enable efficient coded multicasting.