[Paper Review] Storage, Computation, and Communication: A Fundamental Tradeoff in Distributed Computing

Qifa Yan, Sheng Yang+1 more|arXiv (Cornell University)|Jun 20, 2018

Cloud Computing and Resource Management10 references3 citations

TL;DR

This paper establishes the fundamental tradeoff among storage, computation, and communication in distributed computing by deriving a tight converse for the coded distributed computing framework. It proves that the recently proposed D3C scheme achieves the optimal tradeoff, fully characterizing the Pareto-optimal surface of storage, computation, and communication costs in MapReduce-like systems.

ABSTRACT

We consider a MapReduce-like distributed computing system. We derive a lower bound on the communication cost for any given storage and computation costs. This lower bound matches the achievable bound we proposed recently. As a result, we completely characterize the optimal tradeoff between the storage, the computation, and the communication. Our result generalizes the previous one by Li et al. to also account for the number of computed intermediate values.

Motivation & Objective

• To characterize the fundamental tradeoff between storage, computation, and communication in distributed computing systems.
• To extend prior work on coded distributed computing (CDC) by accounting for the actual number of intermediate values computed, not just stored.
• To close the gap between achievable schemes and theoretical limits by proving a matching converse for the D3C scheme.
• To generalize Li et al.'s communication-storage tradeoff to include computation cost as a first-class dimension.
• To provide a complete characterization of the Pareto-optimal tradeoff surface for all three system resources.

Proposed method

• Formalizes a distributed computing model with K nodes and N files, where each node stores a subset of files and computes a subset of intermediate values (IVAs).
• Introduces a new metric for computation cost as the normalized number of IVAs computed per node, distinct from storage cost.
• Uses information-theoretic inequalities and entropy-based bounding techniques to derive a converse bound on communication load.
• Applies an induction-based proof over subsets of nodes to establish a lower bound on conditional entropy, leading to a communication load lower bound.
• Leverages symmetry and independence assumptions between intermediate values to simplify conditional entropy expressions.
• Matches the derived converse bound with the achievable performance of the D3C scheme, proving optimality.

Experimental results

Research questions

• RQ1 What is the fundamental tradeoff between storage, computation, and communication in a distributed computing system?
• RQ2 Can the communication load be minimized for a given combination of storage and computation costs?
• RQ3 Does the D3C scheme achieve the optimal tradeoff, or is there a gap between achievable and converse bounds?
• RQ4 How does accounting for actual computation (i.e., number of IVAs computed) refine the classical storage-communication tradeoff?
• RQ5 Is the communication load lower bound tight, and can it be matched by a practical scheme?

Key findings

• The paper establishes a tight converse bound on the communication load that matches the performance of the D3C scheme, proving optimality.
• The optimal tradeoff surface is completely characterized as the Pareto frontier of all achievable (storage, computation, communication) triples.
• The derived converse bound is expressed as $ T \sum_{j=1}^{s-1} \frac{b_{\mathcal{S},j}}{j} $, where $ b_{\mathcal{S},j} $ counts intermediate values computed by exactly j nodes in subset $ \mathcal{S} $.
• The result generalizes Li et al.'s storage-communication tradeoff by explicitly incorporating computation cost as a design variable.
• The proof relies on an induction-based entropy inequality that holds for all nonempty subsets of nodes, ensuring the bound's generality.
• The key insight is that by allowing selective computation of IVAs, the system can reduce computation cost without increasing communication, achieving better resource tradeoffs.