[Paper Review] Guessing Revisited: A Large Deviations Approach

Manjesh K. Hanawal, Rajesh Sundaresan|NOT FOUND REPOSITORY (Indian Institute of Science Bangalore)|Aug 11, 2010

Computability, Logic, AI Algorithms17 references4 citations

TL;DR

This paper establishes a large deviations framework for the guessing problem, showing that the limiting guessing exponent exists when the information spectrum satisfies the large deviation principle (LDP). The exponent is given by a scalar multiple of the Legendre-Fenchel dual of the rate function, unifying prior results on Rényi entropy rates and compression exponents under exponential cost.

ABSTRACT

The problem of guessing a random string is revisited. A close relation between guessing and compression is first established. Then it is shown that if the sequence of distributions of the information spectrum satisfies the large deviation property with a certain rate function, then the limiting guessing exponent exists and is a scalar multiple of the Legendre-Fenchel dual of the rate function. Other sufficient conditions related to certain continuity properties of the information spectrum are briefly discussed. This approach highlights the importance of the information spectrum in determining the limiting guessing exponent. All known prior results are then re-derived as example applications of our unifying approach.

Motivation & Objective

• To unify existing results on guessing exponents by identifying a general sufficient condition based on large deviations.
• To establish a precise connection between guessing and compression under exponential cost, showing equivalence of their limiting exponents.
• To demonstrate that the limiting guessing exponent equals the Legendre-Fenchel dual of the rate function of the information spectrum, under the LDP.
• To provide a general framework applicable to i.i.d., Markov, and unifilar sources, extending beyond prior specific cases.
• To highlight the role of the information spectrum as the central object governing the limiting behavior of guessing and compression.

Proposed method

• Establish a duality between guessing and compression under exponential cost, showing that the limiting guessing exponent equals the limiting compression exponent.
• Apply Varadhan’s theorem on asymptotics of integrals to the information spectrum $ \frac{1}{n}\ln \frac{1}{P_n(X^n)} $, assuming it satisfies the large deviation principle (LDP).
• Use the Legendre-Fenchel transform to express the limiting guessing exponent as $ \sup_t \{ \beta t - I(t) \} $, where $ I(t) $ is the rate function of the LDP.
• Show that the moment generating function of the information spectrum converges to the normalized Rényi entropy rate, linking the result to known entropy-based measures.
• Verify the technical condition (36) in Varadhan’s theorem by bounding tail probabilities using the cardinality of the alphabet and exponential decay.
• Apply Ky Fan’s minimax result to justify interchanging suprema and infima in the compression formulation, ensuring the existence of optimal length functions.

Experimental results

Research questions

• RQ1 Under what general conditions does the limiting guessing exponent exist for a sequence of distributions?
• RQ2 How is the guessing exponent related to the compression exponent under exponential cost in Campbell’s coding problem?
• RQ3 What is the precise role of the information spectrum $ \frac{1}{n}\ln \frac{1}{P_n(X^n)} $ in determining the limiting guessing exponent?
• RQ4 Can the Legendre-Fenchel dual of the rate function of the information spectrum be used to characterize the limiting guessing exponent?
• RQ5 Which classes of sources (e.g., i.i.d., Markov, unifilar) satisfy the large deviation property for their information spectrum?

Key findings

• The limiting guessing exponent $ E(\rho) = \lim_{n\to\infty} \frac{1}{n} \ln \mathbb{E}[G_n^*(X^n)^\rho] $ exists if the information spectrum satisfies the large deviation principle.
• The limiting exponent is given by $ \beta \cdot \sup_t \{ t - I(t) \} $, where $ \beta = \rho/(1+\rho) $, and $ I(t) $ is the rate function of the LDP.
• The result generalizes known cases: for i.i.d. sources, the exponent reduces to $ \rho H_\alpha(P_1) $ with $ \alpha = 1/(1+\rho) $, matching Rényi entropy.
• For irreducible Markov chains, the exponent equals the logarithm of the Perron-Frobenius eigenvalue of the $ \alpha $-th power of the transition matrix.
• The equivalence between guessing and compression under exponential cost is formally established: both exponents are equal and determined by the same normalized cumulant of the information spectrum.
• All known cases (i.i.d., Markov, unifilar, finite-state sources) satisfy the LDP condition, and thus the limiting exponent exists, as shown via the large deviations framework.