[Paper Review] The Minimum of the Entropy of a Two-Dimensional Distribution with Given Marginals

Giorgio Dall’Aglio, Elisabetta Bona|arXiv (Cornell University)|Apr 8, 2011

Statistical and Computational Modeling1 references3 citations

TL;DR

This paper determines the minimum entropy of a two-dimensional discrete distribution with fixed marginals by constructing a minimizing distribution through row and column rearrangement of the maximum Fréchet distribution function. The key result is that the minimum entropy equals the marginal entropy when the joint distribution concentrates mass along a diagonal, achieving the lowest possible uncertainty under given constraints.

ABSTRACT

The paper search for the minimum of the entropy of a two- dimensional distribution in the Fréchet class, the class of distributions with given marginals. The main result for discrete distributions is an algorithm for building the minimizing distribution, which is given by the maximum distribution function of the Fréchet class after a suitable rearrangement of the rows and of the columns. For absolutely continuous distributions a minimum does not exists, and the infimum is equal to -\infty.

Motivation & Objective

• To identify the distribution within the Fréchet class—i.e., with fixed marginals—that minimizes joint entropy.
• To establish a constructive algorithm for generating the minimizing distribution in the discrete case.
• To analyze the behavior of entropy infimum in the absolutely continuous case, where no minimum exists.
• To characterize conditions under which the minimum joint entropy equals the marginal entropy.

Proposed method

• Proposes an algorithm that rearranges rows and columns of the maximum Fréchet distribution (M(x,y)) to minimize entropy.
• Uses the Fréchet bounds: $ \max\{F(x)+G(y)-1,0\} \leq F(x,y) \leq \min\{F(x),G(y)\} $, with equality at extremes.
• Applies the entropy formula $ H(X,Y) = -\sum_{r,s} p_{r,s} \log p_{r,s} $, with $ p_{r,s} $ constrained by $ \max\{p_{r,\cdot}+p_{\cdot,s}-1,0\} \leq p_{r,s} \leq \min\{p_{r,\cdot},p_{\cdot,s}\} $.
• Constructs the minimizing distribution $ P^* $ by setting $ p_{r,s}^* = p_{r,\cdot} $ when $ r \in I_s $, forming a partition of the support.
• Demonstrates that minimum entropy is achieved when the joint distribution is concentrated along a diagonal, matching the marginal entropy.
• Analyzes the continuous case, showing the infimum of entropy is $ -\infty $, so no minimum exists.

Experimental results

Research questions

• RQ1 What is the minimum possible entropy of a two-dimensional discrete distribution with fixed marginal distributions?
• RQ2 Can a constructive algorithm be developed to generate the distribution that minimizes entropy under fixed marginals?
• RQ3 Under what conditions does the minimum joint entropy equal the marginal entropy?
• RQ4 What happens to the entropy infimum in the case of absolutely continuous distributions with fixed marginals?
• RQ5 How does the structure of the minimizing distribution relate to the Fréchet bounds and the maximum distribution function?

Key findings

• The minimum entropy of a discrete bivariate distribution with fixed marginals is achieved by a distribution that results from rearranging the rows and columns of the maximum Fréchet distribution function.
• The minimizing distribution is constructed such that $ p_{r,s}^* = p_{r,\cdot} $ for $ r \in I_s $, where $ \{I_s\} $ is a partition of the index set satisfying $ \sum_{r \in I_s} p_{r,\cdot} = p_{\cdot,s} $.
• When such a partition exists, the minimum joint entropy equals the marginal entropy: $ H(X^*,Y^*) = H(X) = H(Y^*) $.
• In the case of discrete uniform and geometric marginals, the minimum entropy can be explicitly computed and matches the marginal entropy.
• For absolutely continuous distributions with fixed marginals, the entropy infimum is $ -\infty $, so no minimum exists.
• The minimum entropy can be arbitrarily close to the maximum entropy when one marginal is geometric and the other is uniform, especially as the number of points increases.