[Paper Review] Entropy, Optimization and Counting

Mohit Singh, Nisheeth K. Vishnoi|arXiv (Cornell University)|Apr 30, 2013

Bayesian Modeling and Causal Inference30 references4 citations

TL;DR

This paper establishes an equivalence between counting the size of a discrete combinatorial set and computing its maximum-entropy distribution, showing that efficient algorithms for one problem can be transformed into efficient algorithms for the other. Under mild conditions, max-entropy distributions admit polynomially-sized representations, and the authors provide a polynomial-time ellipsoid-based algorithm that leverages separation oracles and approximate optimization to achieve efficient counting and entropy computation.

ABSTRACT

In this paper we study the problem of computing max-entropy distributions over a discrete set of objects subject to observed marginals. Interest in such distributions arises due to their applicability in areas such as statistical physics, economics, biology, information theory, machine learning, combinatorics and, more recently, approximation algorithms. A key difficulty in computing max-entropy distributions has been to show that they have polynomially-sized descriptions. We show that such descriptions exist under general conditions. Subsequently, we show how algorithms for (approximately) counting the underlying discrete set can be translated into efficient algorithms to (approximately) compute max-entropy distributions. In the reverse direction, we show how access to algorithms that compute max-entropy distributions can be used to count, which establishes an equivalence between counting and computing max-entropy distributions.

Motivation & Objective

• To resolve the fundamental computational challenge of representing and computing max-entropy distributions over exponentially large discrete sets when the set is given implicitly.
• To show that max-entropy distributions admit succinct, polynomial-sized descriptions under general conditions, overcoming the exponential blowup in explicit representation.
• To establish a tight computational equivalence between approximate counting of combinatorial sets and approximate computation of max-entropy distributions.
• To develop a polynomial-time algorithm for estimating the size of a combinatorial set using access to an approximate max-entropy oracle, via the ellipsoid method and separation oracles.
• To demonstrate that efficient optimization over the marginal polytope enables efficient counting, and vice versa, under mild interiority conditions on the marginals.

Proposed method

• Formulate the max-entropy problem as a convex program over the marginal polytope, using duality to reduce the number of variables from |M| to m, the number of elements.
• Use the ellipsoid algorithm with a separation oracle for the η-interior of the marginal polytope P_η(M), ensuring convergence even when the feasible region is not full-dimensional.
• Construct a separation oracle that checks whether a given θ is in the η-interior of P(M) by verifying that the dual solution to the max-entropy program satisfies certain lower bounds.
• Leverage the equivalence between approximate counting and sampling via the Jerrum-Valiant-Vazirani framework to reduce sampling to counting, and vice versa.
• Design a binary search procedure over the logarithm of the size of M, using an approximate max-entropy oracle to test guesses for log|Z|, and refine the estimate using the ellipsoid method.
• Prove correctness by showing that the separating hyperplanes used in the ellipsoid algorithm do not cut off any θ with g(θ) ≥ ζ + ε/16, ensuring the algorithm does not prematurely eliminate the true solution.

Experimental results

Research questions

• RQ1 Can max-entropy distributions over implicitly defined, exponentially large discrete sets be represented with polynomially many bits?
• RQ2 Is there a computational equivalence between approximate counting of combinatorial sets and approximate computation of max-entropy distributions?
• RQ3 Can the ellipsoid method be effectively applied to the marginal polytope for max-entropy computation when the set M is given implicitly?
• RQ4 What conditions ensure that the dual solution to the max-entropy program yields a valid separation oracle for the interior of the marginal polytope?
• RQ5 How can access to an approximate max-entropy oracle be used to achieve a (1±ε)-approximation to the size of the set M?

Key findings

• Max-entropy distributions admit polynomial-sized descriptions via dual variables γ_e, even when the underlying set M is exponentially large, under mild conditions on the marginals.
• The paper establishes a computational equivalence: approximate counting of |M| is possible in polynomial time if and only if approximate max-entropy distributions can be computed in polynomial time.
• An ellipsoid-based algorithm is presented that computes a (1±ε)-approximation to |M| using an approximate max-entropy oracle, with running time polynomial in m and 1/ε.
• The algorithm ensures that the final estimate Z^∘ satisfies (1−ε)|M| ≤ Z^∘ ≤ (1+ε)|M|, with high probability, by maintaining a sequence of shrinking ellipsoids around the true solution.
• The correctness of the algorithm relies on a separation oracle that avoids cutting off points with high g(θ) = log Z_λ, ensuring convergence to a good estimate of log|M|.
• The method works under the assumption that the true marginal vector θ^• lies in the η-interior of P(M), which guarantees that the ellipsoid algorithm converges in polynomial time.