[Paper Review] On Exact and $\infty$-Rényi Common Informations

Lei Yu, Vincent Y. F. Tan|arXiv (Cornell University)|Sep 30, 2018

Wireless Communication Security Techniques44 references3 citations

TL;DR

This paper establishes the equivalence between exact common information and ∞-Rényi common information, proving they are equal for all joint distributions. For doubly symmetric binary sources, it provides a single-letter characterization showing both are strictly larger than Wyner’s common information, resolving an open problem posed by Kumar, Li, and El Gamal.

ABSTRACT

Recently, two extensions of Wyner's common information extemdash exact and Rényi common informations extemdash were introduced respectively by Kumar, Li, and El Gamal (KLE), and the present authors. The class of common information problems involves determining the minimum rate of the common input to two independent processors needed to exactly or approximately generate a target joint distribution. For the exact common information problem, exact generation of the target distribution is required, while for Wyner's and $α$-Rényi common informations, the relative entropy and Rényi divergence with order $α$ were respectively used to quantify the discrepancy between the synthesized and target distributions. The exact common information is larger than or equal to Wyner's common information. However, it was hitherto unknown whether the former is strictly larger than the latter for some joint distributions. In this paper, we first establish the equivalence between the exact and $\infty$-Rényi common informations, and then provide single-letter upper and lower bounds for these two quantities. For doubly symmetric binary sources, we show that the upper and lower bounds coincide, which implies that for such sources, the exact and $\infty$-Rényi common informations are completely characterized. Interestingly, we observe that for such sources, these two common informations are strictly larger than Wyner's. This answers an open problem posed by KLE. Furthermore, we extend Wyner's, $\infty$-Rényi, and exact common informations to sources with countably infinite or continuous alphabets, including Gaussian sources.

Motivation & Objective

• To resolve the open question of whether exact common information is strictly larger than Wyner’s common information for any joint distribution.
• To establish a single-letter characterization of exact and ∞-Rényi common informations for general sources.
• To extend the definitions of common information to sources with countably infinite or continuous alphabets, including Gaussian sources.
• To demonstrate that for doubly symmetric binary sources, the exact and ∞-Rényi common informations are completely characterized and strictly exceed Wyner’s common information.

Proposed method

• Introduces ∞-Rényi common information as the minimum common rate when the discrepancy is measured by Rényi divergence of order ∞.
• Derives a single-letter expression for ∞-Rényi common information as the minimum over all product distributions of the Rényi divergence of order ∞.
• Proves that the exact common information equals the ∞-Rényi common information by showing both are characterized by the same optimization problem.
• Uses a decomposition of the target distribution π_XY into a mixture of product distributions to construct a valid coupling that achieves the bound.
• Applies the min-max property of Rényi divergence of order ∞ to derive tight upper and lower bounds.
• Establishes additivity of the ∞-Rényi common information over i.i.d. extensions, enabling the asymptotic characterization.

Experimental results

Research questions

• RQ1 Is the exact common information strictly larger than Wyner’s common information for any joint distribution?
• RQ2 Can the exact common information be characterized in closed form for specific sources, such as doubly symmetric binary sources?
• RQ3 Is there an equivalence between exact common information and ∞-Rényi common information?
• RQ4 How do these common information measures behave for sources with infinite or continuous alphabets?
• RQ5 Can the single-letter characterization of ∞-Rényi common information be used to derive tight bounds on exact common information?

Key findings

• The exact common information is exactly equal to the ∞-Rényi common information for all joint distributions.
• For doubly symmetric binary sources, the exact and ∞-Rényi common informations are completely characterized by a single-letter formula.
• For doubly symmetric binary sources, both the exact and ∞-Rényi common informations are strictly larger than Wyner’s common information.
• The ∞-Rényi common information is given by the minimum Rényi divergence of order ∞ between any product distribution and the target distribution π_XY.
• The exact common information is characterized as the limit of the common entropy of i.i.d. extensions, and equals the ∞-Rényi common information.
• The results extend to sources with countably infinite or continuous alphabets, including Gaussian sources, via appropriate generalizations of the divergence measures.