[Paper Review] Efficient multivariate entropy estimation via $k$-nearest neighbour distances

Thomas B. Berrett, Richard J. Samworth+1 more|arXiv (Cornell University)|Jun 1, 2016

Statistical Methods and Inference18 references15 citations

TL;DR

This paper proposes a weighted k-nearest neighbor estimator for multivariate differential entropy that achieves asymptotic efficiency in arbitrary dimensions by canceling leading bias terms through optimal weight selection. The method generalizes the Kozachenko-Leonenko estimator, enabling efficient entropy estimation under weaker smoothness conditions and unbounded support, with asymptotically valid confidence intervals of minimal width.

ABSTRACT

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper, we seek entropy estimators that are efficient and achieve the local asymptotic minimax lower bound with respect to squared error loss. To this end, we study weighted averages of the estimators originally proposed by Kozachenko and Leonenko (1987), based on the $k$-nearest neighbour distances of a sample of $n$ independent and identically distributed random vectors in $\mathbb{R}^d$. A careful choice of weights enables us to obtain an efficient estimator in arbitrary dimensions, given sufficient smoothness, while the original unweighted estimator is typically only efficient when $d \leq 3$. In addition to the new estimator proposed and theoretical understanding provided, our results facilitate the construction of asymptotically valid confidence intervals for the entropy of asymptotically minimal width.

Motivation & Objective

• To develop an efficient, asymptotically optimal entropy estimator for multivariate distributions in arbitrary dimensions.
• To overcome the inefficiency of the standard Kozachenko-Leonenko estimator in high dimensions (d ≥ 4) due to non-trivial bias.
• To construct asymptotically valid confidence intervals for entropy with minimal width.
• To extend efficient entropy estimation to densities with unbounded support, a setting where prior methods fail.

Proposed method

• Proposes a weighted average of Kozachenko-Leonenko estimators using k-nearest neighbor distances for different k values.
• Derives optimal weights to cancel the dominant bias terms in the asymptotic expansion of the estimator.
• Uses a second-order Taylor expansion of entropy around a density estimator to analyze bias and variance.
• Imposes smoothness conditions on the density and its derivatives to ensure the validity of the asymptotic expansion.
• Applies H"older's inequality and Taylor series approximations on sets where the score function is small to control error terms.
• Establishes asymptotic normality and efficiency by showing convergence to the local asymptotic minimax lower bound under regularity conditions.

Experimental results

Research questions

• RQ1 Can a weighted k-NN estimator achieve asymptotic efficiency in multivariate entropy estimation for arbitrary dimensions d?
• RQ2 What choice of weights on k-NN distances eliminates the dominant bias term in high-dimensional settings?
• RQ3 Does the proposed estimator remain efficient when the density has unbounded support, unlike prior methods?
• RQ4 Can asymptotically valid confidence intervals of minimal width be constructed using this estimator?
• RQ5 How does the estimator's performance compare to the unweighted Kozachenko-Leonenko estimator in high dimensions?

Key findings

• The proposed weighted k-NN estimator achieves asymptotic efficiency in arbitrary dimensions d, unlike the unweighted estimator which is only efficient for d ≤ 3.
• The estimator attains the local asymptotic minimax lower bound for squared error loss under sufficient smoothness conditions.
• The method enables the construction of asymptotically valid confidence intervals for entropy with minimal width, a key practical advantage.
• The estimator remains efficient even when the density has unbounded support, a significant extension beyond prior work that required compact support and bounded density away from zero.
• The asymptotic distribution of the estimator is normal with variance equal to the Fisher information, confirming efficiency.
• The optimal weights are derived analytically to cancel the leading bias term, and their existence is proven under mild smoothness assumptions.