[Paper Review] Information Inequalities for Joint Distributions, with Interpretations and Applications

Mokshay Madiman, Prasad Tetali|arXiv (Cornell University)|Dec 31, 2008

Wireless Communication Security Techniques38 references4 citations

TL;DR

This paper introduces a unified framework of information inequalities for joint entropies using submodular function theory, generalizing Shannon's chain rule, Han's, Fujishige's, and Shearer's inequalities. It establishes tight upper and lower bounds on joint entropy in terms of subset entropies via fractional covers and packings, with applications to combinatorics, matrix determinants, and hypothesis testing.

ABSTRACT

Upper and lower bounds are obtained for the joint entropy of a collection of random variables in terms of an arbitrary collection of subset joint entropies. These inequalities generalize Shannon's chain rule for entropy as well as inequalities of Han, Fujishige and Shearer. A duality between the upper and lower bounds for joint entropy is developed. All of these results are shown to be special cases of general, new results for submodular functions-- thus, the inequalities presented constitute a richly structured class of Shannon-type inequalities. The new inequalities are applied to obtain new results in combinatorics, such as bounds on the number of independent sets in an arbitrary graph and the number of zero-error source-channel codes, as well as new determinantal inequalities in matrix theory. A new inequality for relative entropies is also developed, along with interpretations in terms of hypothesis testing. Finally, revealing connections of the results to literature in economics, computer science, and physics are explored.

Motivation & Objective

• To generalize classical entropy inequalities such as Shannon's chain rule, Han's inequality, and Shearer's lemma into a single, coherent framework.
• To establish new upper and lower bounds for joint entropy based on arbitrary collections of subsets, using degrees of coverage and fractional hypergraph theory.
• To demonstrate that these inequalities arise naturally from submodular function theory, thereby unifying diverse results in information theory and combinatorics.
• To apply the derived inequalities to new problems in combinatorics (e.g., counting independent sets and zero-error codes), matrix determinants, and relative entropy in hypothesis testing.
• To clarify the structural limitations of entropy by showing that the conditional entropy chain rule function is not submodular, thus distinguishing deeper constraints beyond submodularity.

Proposed method

• Develops fractional coverings and packings using hypergraphs to model subset dependencies in joint entropy inequalities.
• Introduces a general inequality for submodular set functions, which serves as the foundation for deriving all subsequent entropy inequalities.
• Derives two main forms: the strong fractional form and the strong degree form, generalizing the weak degree form in Proposition I.
• Applies the general submodular inequality to entropy functions, yielding bounds on $ H(X_{[n]}) $ in terms of $ H(X_s) $ and $ H(X_s|X_{s^c}) $, weighted by subset degrees.
• Uses duality between upper and lower bounds to reveal structural symmetry in entropy inequalities.
• Applies the results to combinatorics via entropy-based counting, such as bounding the number of independent sets in graphs and zero-error source-channel codes.

Experimental results

Research questions

• RQ1 Can classical entropy inequalities such as Han’s and Shearer’s be unified under a single theoretical framework?
• RQ2 What are the tightest possible upper and lower bounds on joint entropy in terms of subset entropies for arbitrary collections of subsets?
• RQ3 How do fractional hypergraph covers and packings relate to information-theoretic inequalities and submodular functions?
• RQ4 To what extent can entropy functions be characterized beyond submodularity, and what are the implications for information-theoretic bounds?
• RQ5 What new combinatorial and matrix-theoretic results can be derived from these generalized entropy inequalities?

Key findings

• The paper establishes a new class of information inequalities that generalize Shannon’s chain rule, Han’s inequality, and Shearer’s lemma as special cases.
• A duality is revealed between upper and lower bounds for joint entropy, with the bounds expressed in terms of the minimal and maximal degrees of indices in a subset collection.
• The main inequality is derived as a corollary of a general result on submodular functions, providing a unified and powerful theoretical foundation.
• The bounds are tight and apply to both discrete and continuous (differential) entropy, with the continuous case requiring fractional partitions.
• The results yield new combinatorial bounds: for example, on the number of independent sets in an arbitrary graph and the number of zero-error source-channel codes.
• A counterexample is provided showing that the conditional entropy chain rule function $ \bar{\tt e}(s) = H(X_s|X_{<s}) $ is not submodular, highlighting the limits of submodular approximation in entropy.