[Paper Review] The Benefit of Being Flexible in Distributed Computation

Linqi Song, Sundara Rajan Srinivasavaradhan+1 more|arXiv (Cornell University)|May 23, 2017

Distributed systems and fault tolerance3 citations

TL;DR

This paper demonstrates that flexibility in assigning computation functions to nodes in a distributed system with random data placement can drastically reduce communication costs. By strategically assigning functions to nodes that already possess the required data, the system can eliminate data shuffling entirely when data observation probability exceeds a critical threshold, achieving near-zero communication in high-coverage regimes.

ABSTRACT

In wireless distributed computing, networked nodes perform intermediate computations over data placed in their memory and exchange these intermediate values to calculate function values. In this paper we consider an asymmetric setting where each node has access to a random subset of the data, i.e., we cannot control the data placement. The paper makes a simple point: we can realize significant benefits if we are allowed to be "flexible", and decide which node computes which function, in our system. We make this argument in the case where each function depends on only two of the data messages, as is the case in similarity searches. We establish a percolation in the behavior of the system, where, depending on the amount of observed data, by being flexible, we may need no communication at all.

Motivation & Objective

• To investigate the communication cost trade-offs in distributed computing under random data placement.
• To analyze whether flexible assignment of functions to nodes can reduce communication compared to fixed assignment.
• To identify conditions under which data shuffling becomes unnecessary due to inherent data availability.
• To quantify the benefits of function assignment flexibility in terms of communication reduction, especially in asymmetric function-data relationships.

Proposed method

• Models a distributed computing system with m nodes, each holding a random subset of K data messages.
• Represents function-data dependencies as a random bipartite graph B(m, n, p), where p is the probability a node holds a given data message.
• Analyzes communication cost under two schemes: fixed function assignment (where functions are pre-assigned) and flexible function assignment (where functions are assigned to nodes with available data).
• Uses percolation theory and probabilistic bounds (e.g., A.M-G.M, Chernoff) to derive thresholds for when communication becomes unnecessary.
• Evaluates both uncoded and coded transmission schemes, focusing on the number of transmissions required.
• Derives a threshold probability pth = 1 − (log K / K)^{1/C} above which flexible assignment can eliminate data shuffling.

Experimental results

Research questions

• RQ1 Under random data placement, what is the minimum communication cost achievable in distributed function computation?
• RQ2 How does flexible assignment of functions to nodes affect communication cost compared to fixed assignment?
• RQ3 At what data observation probability does the system transition from requiring significant communication to needing none?
• RQ4 Can coded or uncoded transmissions be reduced to zero when nodes are allowed to choose which function to compute based on local data?
• RQ5 What is the impact of function dependency structure (e.g., each function depends on two data messages) on communication efficiency?

Key findings

• When the data observation probability p exceeds the threshold pth = 1 − (log K / K)^{1/C}, flexible function assignment can eliminate data shuffling entirely, requiring zero communication.
• For fixed function assignment, communication cost remains Θ(K) even with coding when p is below the threshold, indicating no coding gain.
• With flexible assignment, communication cost drops to zero almost surely when p > 1 − o(1), particularly when p > 1 − log K / K.
• For uncoded transmissions, the expected number of transmissions is K(2 − 2p + p²), and this drops significantly when p is high.
• The benefit of flexibility is most pronounced in the regime where p ∈ (log K / n)^{ω(1)} to 1 − log K / K, where fixed assignment needs Θ(K) transmissions while flexible assignment needs none.
• The threshold pth = 1 − (log K / K)^{1/C} is derived using probabilistic inequalities and percolation analysis, showing a sharp transition in system behavior.