[Paper Review] Minimax rates of entropy estimation on large alphabets via best polynomial approximation

Yihong Wu, Pengkun Yang|arXiv (Cornell University)|Jul 1, 2014

Machine Learning and Algorithms49 references17 citations

TL;DR

This paper establishes the minimax rate for Shannon entropy estimation on large alphabets using best polynomial approximation, showing that the optimal risk scales as $\left(\frac{k}{n\log k}\right)^2 + \frac{\log^2 k}{n}$ when $n \gtrsim \frac{k}{\log k}$, and proving the existence of a linear-time estimator achieving this bound. The analysis relies on duality between estimation and approximation theory to derive tight upper and lower bounds.

ABSTRACT

Consider the problem of estimating the Shannon entropy of a distribution over $k$ elements from $n$ independent samples. We show that the minimax mean-square error is within universal multiplicative constant factors of $$\Big(\frac{k }{n \log k}\Big)^2 + \frac{\log^2 k}{n}$$ if $n$ exceeds a constant factor of $\frac{k}{\log k}$; otherwise there exists no consistent estimator. This refines the recent result of Valiant-Valiant \cite{VV11} that the minimal sample size for consistent entropy estimation scales according to $Θ(\frac{k}{\log k})$. The apparatus of best polynomial approximation plays a key role in both the construction of optimal estimators and, via a duality argument, the minimax lower bound.

Motivation & Objective

• To characterize the fundamental minimax risk for estimating Shannon entropy over large alphabets from i.i.d. samples.
• To establish sharp upper and lower bounds on the minimax mean-square error in terms of sample size $n$ and alphabet size $k$.
• To develop a computationally efficient, linear-time estimator that achieves the minimax risk within universal constant factors.
• To bridge information theory and approximation theory by using best polynomial approximation as a central analytical tool.

Proposed method

• The minimax risk is analyzed through duality with the best polynomial approximation of the entropy function $f(x) = -x\log x$ on $[0,1]$.
• The authors use the theory of best polynomial approximation to derive bounds on the bias of entropy estimators, particularly in the high-dimensional, undersampled regime.
• A key technical component is the use of Chebyshev alternation theorem to analyze endpoint approximation errors in polynomial approximation.
• The minimax lower bound is derived via a duality argument between estimation risk and approximation error in the $L^\infty$ norm.
• A novel linear-time estimator is constructed based on the best polynomial approximation, which matches the minimax risk up to universal constants.
• The analysis distinguishes two regimes: one dominated by bias ($n \lesssim k^2 / \log^4 k$) and one by variance ($n \gtrsim k^2 / \log^4 k$).

Experimental results

Research questions

• RQ1 What is the fundamental minimax rate for estimating Shannon entropy when the alphabet size $k$ is large and the sample size $n$ is sublinear in $k$?
• RQ2 Can the minimax risk be characterized within universal constant factors, and what is its exact functional dependence on $n$ and $k$?
• RQ3 Is there a computationally efficient estimator that achieves the minimax risk up to a constant factor?
• RQ4 How does the theory of best polynomial approximation relate to the minimax risk in entropy estimation?

Key findings

• The minimax risk for entropy estimation is $R^*(k,n) \asymp \left(\frac{k}{n\log k}\right)^2 + \frac{\log^2 k}{n}$ when $n \gtrsim \frac{k}{\log k}$.
• When $n \lesssim \frac{k}{\log k}$, no consistent estimator exists, as the minimax risk is bounded away from zero.
• The bias term $\left(\frac{k}{n\log k}\right)^2$ dominates when $n \lesssim \frac{k^2}{\log^4 k}$, reflecting the challenge of undersampling in large alphabets.
• The variance term $\frac{\log^2 k}{n}$ corresponds to the classical parametric rate governed by the central limit theorem.
• A linear-time estimator is constructed that achieves the minimax risk within universal constant factors, making it computationally practical.
• The use of best polynomial approximation provides a tight duality between estimation risk and approximation error, enabling sharp bounds.