[Paper Review] Communication Requirements for Generating Correlated Random Variables

Paul Cuff|ArXiv.org|May 1, 2008

Bayesian Modeling and Causal Inference4 references13 citations

TL;DR

This paper characterizes the tradeoff between common randomness and description rate for simulating a discrete memoryless channel, showing that mutual information is sufficient when common randomness is available, while Wyner's common information is required when no such randomness exists. The optimal rate region is derived using information-theoretic techniques, resolving a long-standing tension between Wyner's and Bennett-Shor's results.

ABSTRACT

Two familiar notions of correlation are rediscovered as extreme operating points for simulating a discrete memoryless channel, in which a channel output is generated based only on a description of the channel input. Wyner's "common information" coincides with the minimum description rate needed. However, when common randomness independent of the input is available, the necessary description rate reduces to Shannon's mutual information. This work characterizes the optimal tradeoff between the amount of common randomness used and the required rate of description.

Motivation & Objective

• To resolve the apparent contradiction between Wyner's common information and the Bennett-Shor reverse Shannon theorem in channel simulation.
• To characterize the optimal tradeoff between the rate of common randomness and the description rate needed to simulate a discrete memoryless channel.
• To establish that mutual information is sufficient for channel simulation when common randomness is available, and common information when it is not.
• To provide a framework applicable to cooperative game theory, particularly in repeated games with limited communication.

Proposed method

• Formulates a channel simulation problem where an encoder describes an i.i.d. source to a decoder that generates a correlated output using a description and possibly common randomness.
• Uses total variation distance as the metric for statistical indistinguishability between the simulated and real channel output distributions.
• Derives the optimal rate region by analyzing joint distributions over sequences and auxiliary random variables, leveraging Markov chains and typical sequences.
• Introduces a coding scheme based on random codebooks of auxiliary sequences (U^n) that are jointly typical with X^n, enabling the decoder to generate Y^n conditionally.
• Applies the data processing inequality and mutual information decomposition to derive converse bounds, showing that the common information of the joint distribution is the minimal rate when no common randomness is available.
• Uses a time-sharing random variable W to unify strategies with varying correlation levels, enabling the derivation of the optimal rate for a given payoff in game-theoretic settings.

Experimental results

Research questions

• RQ1 What is the minimal description rate required to simulate a discrete memoryless channel when no common randomness is available?
• RQ2 How does the availability of common randomness reduce the required description rate in channel simulation?
• RQ3 Can the tradeoff between common randomness rate and description rate be characterized precisely for a given joint distribution?
• RQ4 How does this channel simulation framework relate to game-theoretic coordination in repeated games with limited communication?
• RQ5 What is the operational significance of Wyner's common information versus mutual information in channel simulation?

Key findings

• When no common randomness is available, the minimal description rate required to simulate a discrete memoryless channel is equal to Wyner's common information C(X;Y).
• When common randomness is available at rate R2, the required description rate R1 can be reduced to the mutual information I(X;Y), achieving the reverse Shannon theorem regime.
• The optimal tradeoff between common randomness rate R2 and description rate R1 is characterized as a convex region, with mutual information and common information as extreme points.
• In a repeated game setting with limited secure communication, the minimal communication rate needed to achieve a given payoff is equal to the minimum common information over all strategies achieving that payoff.
• The converse proof shows that any achievable rate must satisfy H(U) ≥ I(X^n, Y^n; U), which leads to a lower bound involving the common information when averaged over time.
• The achievability scheme uses random codebooks of auxiliary sequences (U^n) that are jointly typical with X^n, enabling the decoder to generate Y^n conditionally based on U^n and the description.