[Paper Review] On the Kozachenko-Leonenko entropy estimator

Nicolas Fournier, Sylvain Delattre|arXiv (Cornell University)|Feb 24, 2016

Statistical Methods and Inference12 references3 citations

TL;DR

This paper provides a rigorous non-asymptotic analysis of the Kozachenko-Leonenko entropy estimator, deriving its bias and variance in general dimensions. It establishes a central limit theorem in dimensions 1 and 2 and uses Richardson extrapolation to construct a bias-corrected estimator with explicit asymptotic confidence intervals in any dimension.

ABSTRACT

We study in details the bias and variance of the entropy estimator proposed by Kozachenko and Leonenko for a large class of densities on $\mathbb{R}^d$. We then use the work of Bickel and Breiman to prove a central limit theorem in dimensions $1$ and $2$. In higher dimensions, we provide a development of the bias in terms of powers of $N^{-2/d}$. This allows us to use a Richardson extrapolation to build, in any dimension, an estimator satisfying a central limit theorem and for which we can give some some explicit (asymptotic) confidence intervals.

Motivation & Objective

• To provide a detailed non-asymptotic study of the bias and variance of the Kozachenko-Leonenko entropy estimator for general densities on R^d.
• To establish a central limit theorem for the estimator in dimensions 1 and 2 using Bickel and Breiman's framework.
• To develop a higher-order bias expansion in terms of N^{-2/d} for arbitrary dimensions.
• To construct a bias-corrected estimator via Richardson extrapolation that satisfies a central limit theorem and admits explicit asymptotic confidence intervals in any dimension.

Proposed method

• The estimator is defined as $ H_N = \frac{1}{N+1}\sum_{i=1}^{N+1}\log Y^N_i + \gamma + \log v_d $, where $ Y^N_i = N(R^N_i)^d $, with $ R^N_i $ being the distance from $ X_i $ to its nearest neighbor among $ X_1, \dots, X_{N+1} $.
• The authors use a heuristic argument based on the approximate exponential distribution of $ Y^N_i $ given $ X_i $, with rate $ v_d f(X_i) $, to justify the consistency of $ H_N $.
• For bias analysis, they derive a series expansion in powers of $ N^{-2/d} $, valid under regularity conditions on the density $ f $, including polynomial and exponential tails.
• They apply Bickel and Breiman's results to prove asymptotic normality in dimensions 1 and 2, under moment conditions on $ \log f $ and $ \log^2 f $.
• For higher dimensions, they use Richardson extrapolation: by computing $ H_N $ at multiple sample sizes, they eliminate the leading bias term $ c N^{-2/d} $, yielding a faster-converging estimator.
• The resulting extrapolated estimator satisfies a central limit theorem and allows for explicit asymptotic confidence intervals via the normal approximation.

Experimental results

Research questions

• RQ1 What is the precise bias structure of the Kozachenko-Leonenko entropy estimator in arbitrary dimensions?
• RQ2 Under what conditions does the estimator satisfy a central limit theorem in dimensions 1 and 2?
• RQ3 Can the bias of the estimator be systematically corrected in higher dimensions using extrapolation?
• RQ4 What are the explicit asymptotic confidence intervals for the entropy estimator after bias correction?
• RQ5 How do the regularity and tail behavior of the density $ f $ affect the convergence rate and bias expansion?

Key findings

• The bias of the Kozachenko-Leonenko estimator admits a power series expansion in $ N^{-2/d} $, with leading term $ c N^{-2/d} $, valid for densities in a broad class including those with polynomial and exponential tails.
• In dimensions 1 and 2, the estimator satisfies a central limit theorem under moment conditions on $ \log f $ and $ \log^2 f $, as established via Bickel and Breiman's framework.
• For dimensions $ d \geq 3 $, the central limit theorem does not hold for the raw estimator due to bias dominance, but Richardson extrapolation removes the leading bias term.
• The extrapolated estimator achieves $ \sqrt{N} $-consistency and asymptotic normality in any dimension, enabling the construction of explicit asymptotic confidence intervals.
• The method applies to densities with heavy tails (e.g., $ f(x) \sim |x|^{-a} e^{-|x|} $) and bounded densities with singularities (e.g., beta-like densities), provided the regularity parameter $ \tau $ satisfies $ \tau \geq 2 $ in dimension $ d \geq 3 $.
• The paper provides sufficient conditions for the applicability of the central limit theorem via Corollary 7 and Corollary 9, depending on dimension and tail behavior, with explicit constraints on the exponent $ a $ and dimension $ d $.