[Paper Review] Profile Entropy: A Fundamental Measure for the Learnability and Compressibility of Discrete Distributions

Hao Yi, Alon Orlitsky|arXiv (Cornell University)|Feb 26, 2020

Diffusion and Search Dynamics53 references4 citations

TL;DR

This paper introduces profile entropy as a fundamental measure that unifies estimation, inference, and compression for discrete distributions. It shows that profile entropy determines the learnability and compressibility of distributions, with optimal algorithms achieving performance competitive to the best possible estimators and compressors, all adaptively based on the data's intrinsic complexity as captured by its profile entropy.

ABSTRACT

The profile of a sample is the multiset of its symbol frequencies. We show that for samples of discrete distributions, profile entropy is a fundamental measure unifying the concepts of estimation, inference, and compression. Specifically, profile entropy a) determines the speed of estimating the distribution relative to the best natural estimator; b) characterizes the rate of inferring all symmetric properties compared with the best estimator over any label-invariant distribution collection; c) serves as the limit of profile compression, for which we derive optimal near-linear-time block and sequential algorithms. To further our understanding of profile entropy, we investigate its attributes, provide algorithms for approximating its value, and determine its magnitude for numerous structural distribution families.

Motivation & Objective

• To identify a fundamental measure that captures the intrinsic complexity of discrete distributions for estimation, inference, and compression.
• To unify the concepts of distribution estimation, symmetric property inference, and profile compression under a single theoretical framework.
• To develop adaptive algorithms whose performance guarantees depend directly on profile entropy, achieving near-optimality relative to the best possible estimators and compressors.
• To characterize the magnitude of profile entropy for structured families such as log-concave, power-law, and histogram distributions.
• To design optimal near-linear-time algorithms for profile compression, both block and sequential.

Proposed method

• The paper defines the profile of a sample as the multiset of symbol frequencies, and introduces profile entropy as the Shannon entropy of this multiset.
• It uses Poisson sampling to simplify the analysis of profile distributions and derives concentration bounds for profile dimension and entropy.
• A simplified proxy, $ H^{ ext{S}}_n(p) $, is introduced to approximate profile entropy, enabling tighter analysis of its magnitude.
• Theoretical guarantees are derived using tools from probability, including the median trick and bounds on the volume of discrete ellipsoids in high dimensions.
• Optimal compression algorithms are constructed using profile-based coding schemes, with block and sequential variants achieving near-optimal compression rates.
• The paper leverages PML (Profile Maximum Likelihood) and sorted distribution estimation to connect profile entropy to symmetric property estimation and uniformity testing.

Experimental results

Research questions

• RQ1 How does profile entropy relate to the speed of distribution estimation relative to the best natural estimator?
• RQ2 To what extent can symmetric properties of discrete distributions be inferred efficiently, and how does profile entropy characterize this rate?
• RQ3 What is the fundamental limit of profile compression, and can optimal algorithms be designed that achieve this limit?
• RQ4 How does profile entropy behave for structured distribution families such as log-concave, power-law, and histogram distributions?
• RQ5 Can profile entropy be efficiently approximated, and what are its concentration properties?

Key findings

• Profile entropy determines the competitive ratio of distribution estimation: algorithms can achieve performance within a constant factor of the best possible estimator, with guarantees depending on profile entropy.
• Profile entropy characterizes the rate of inference for symmetric properties, with optimal estimators achieving performance competitive to the best label-invariant estimators over any distribution collection.
• Profile compression is fundamentally limited by profile entropy, and the paper presents optimal near-linear-time block and sequential compression algorithms that achieve this limit.
• For log-concave and power-law distributions, profile entropy is bounded by $ O(\log n) $, indicating low intrinsic complexity and efficient learnability.
• The proxy $ H^{ ext{S}}_n(p) $ provides a nearly-tight approximation to the expected profile entropy, enabling precise characterization of its magnitude across distribution families.
• For multivariate discrete Gaussians, profile entropy is bounded by $ \mathcal{O}(\log n) \left(1 + \min\left\{ \frac{n}{C}, \gamma_d (\alpha_\Sigma \cdot \beta_{d,n})^d C \right\} \right) $, where $ C $ is a normalization constant related to the distribution's support.