[Paper Review] A Unified Framework for Problems on Guessing, Source Coding and Task Partitioning

M. Ashok Kumar, Albert Sunny+2 more|arXiv (Cornell University)|Jul 16, 2019

Wireless Communication Security Techniques16 references4 citations

TL;DR

This paper presents a unified optimization framework that connects source coding, guessing, and task partitioning problems through Rényi entropy and Sundaresan's divergence. By generalizing Campbell’s cumulant minimization and Arikan’s guessing problems, the authors derive lower bounds using Rényi entropy and extend results to mismatched settings, revealing deep structural similarities across these information-theoretic problems.

ABSTRACT

We study four problems namely, Campbell's source coding problem, Arikan's guessing problem, Huieihel et al.'s memoryless guessing problem, and Bunte and Lapidoth's task partitioning problem. We observe a close relationship among these problems. In all these problems, the objective is to minimize moments of some functions of random variables, and Rényi entropy and Sundaresan's divergence arise as optimal solutions. This motivates us to establish a connection among these four problems. In this paper, we study a more general problem and show that Rényi and Shannon entropies arise as its solution. We show that the problems on source coding, guessing and task partitioning are particular instances of this general optimization problem, and derive the lower bounds using this framework. We also refine some known results and present new results for mismatched version of these problems using a unified approach. We strongly feel that this generalization would, in addition to help in understanding the similarities and distinctiveness of these problems, also help to solve any new problem that falls in this framework.

Motivation & Objective

• To unify seemingly disparate problems in information theory—source coding, guessing, and task partitioning—under a single mathematical optimization framework.
• To demonstrate that Rényi entropy and Sundaresan’s divergence emerge naturally as optimal solutions across these problems.
• To refine known results and derive new bounds for mismatched versions of these problems using the proposed generalization.
• To provide a foundation for solving new problems falling within this unified framework by highlighting structural similarities and distinctions.

Proposed method

• Formulate a general optimization problem minimizing moments of functions of random variables, with Rényi entropy as the optimal solution.
• Apply Campbell’s cumulant minimization principle to derive bounds for source coding, extending it to mismatched settings using Sundaresan’s divergence.
• Use the duality between Rényi entropy and relative α-Rényi entropy (Sundaresan’s divergence) to unify lower bounds across problems.
• Leverage properties of Rényi entropy, including its limit behavior as α→1 (reducing to Shannon entropy), to unify asymptotic results.
• Construct partition functions and use probabilistic inequalities (e.g., Hölder’s inequality) to derive bounds on moments of guessing and partitioning functions.
• Apply variational techniques and optimization over probability distributions to prove tightness of bounds and asymptotic convergence.

Experimental results

Research questions

• RQ1 How can source coding, guessing, and task partitioning problems be unified under a single optimization framework?
• RQ2 What role does Rényi entropy play in minimizing moments of code lengths, guessing counts, and partition functions?
• RQ3 How do mismatched distributions affect the performance bounds in these problems, and can they be quantified using relative α-Rényi entropy?
• RQ4 Can the asymptotic behavior of guessing and coding moments be characterized using Rényi entropy in the limit of long sequences?
• RQ5 What are the implications of this framework for new problems in information theory, particularly in continuous or infinite state spaces?

Key findings

• The general optimization framework unifies source coding, guessing, and task partitioning, with Rényi entropy emerging as the optimal solution for minimizing moments of functions of random variables.
• For the guessing problem, the paper establishes that the expected ρ-th moment of the number of guesses satisfies limₙ→∞𝔼[Aₙ(Xⁿ)ᵖ] = 1 when log N > Hₐ(P) and ρ > 0, indicating asymptotic optimality.
• When log N < Hₐ(P), the ρ-th moment diverges to infinity as n→∞, showing a sharp threshold in performance depending on the alphabet size and Rényi entropy.
• The mismatched case is analyzed using Sundaresan’s divergence, showing that the penalty for mismatched distribution is captured by Iₐ(P,Q), extending Campbell’s result to general α.
• The framework provides tight lower bounds for all three problems, with Rényi entropy Hₐ(P) serving as the fundamental limit in each case.
• The paper proves that the optimal partition function A(x) can be expressed as A(x) = Z_Q,α / (N Q(x)^α), linking the structure of optimal solutions across problems.