[Paper Review] Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

Dávid Pál, Barnabás Póczos+1 more|arXiv (Cornell University)|Mar 9, 2010

Bayesian Methods and Mixture Models38 references92 citations

TL;DR

This paper proposes a novel, nonparametric estimator for Rényi entropy and mutual information using generalized nearest-neighbor graphs and empirical copulas. It establishes almost sure consistency and provides finite-sample high-probability error bounds under Lipschitz density assumptions, marking the first rate-of-convergence analysis for Rényi entropy estimation using k-NN graphs.

ABSTRACT

We present simple and computationally efficient nonparametric estimators of Rényi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are calculated as the sum of $p$-th powers of the Euclidean lengths of the edges of the `generalized nearest-neighbor' graph of the sample and the empirical copula of the sample respectively. For the first time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis.

Motivation & Objective

• To develop a computationally efficient, nonparametric estimator of Rényi entropy and mutual information without relying on density estimation.
• To correct and rigorously prove the almost sure consistency of k-NN-based Rényi entropy estimators, addressing flaws in prior proofs.
• To establish the first finite-sample high-probability error bounds (rates of convergence) for Rényi entropy estimation under Lipschitz density conditions.
• To extend the k-NN graph approach to mutual information estimation via empirical copula transformation, proving consistency for d ≥ 3 and α ∈ (1/2, 1).
• To demonstrate the computational advantages of k-NN graphs over MST and TSP for multiple α values due to reusability and parallelizability.

Proposed method

• Estimates Rényi entropy using the sum of p-th powers of edge lengths in a generalized k-NN graph, where each point connects to an arbitrary subset of its k nearest neighbors.
• Employs the empirical copula transformation to map the original i.i.d. sample into the unit cube, enabling mutual information estimation via graph structure on transformed data.
• Uses a perturbation analysis to bound the difference between the true and empirical graph weight sums, relying on Lipschitz continuity and triangle inequality for distance differences.
• Applies a Borel-Cantelli argument with δ = 1/n² to establish almost sure consistency of the entropy estimator under bounded support and α ∈ (0,1).
• Derives high-probability error bounds using concentration inequalities and the perturbation lemma, with different rates depending on p (i.e., α = 1 - p/d).
• Proves consistency of the mutual information estimator by combining the copula transformation with the generalized k-NN graph structure, leveraging known results on copula convergence.

Experimental results

Research questions

• RQ1 Can a k-NN graph-based estimator achieve almost sure consistency for Rényi entropy estimation, and can this be rigorously proven without relying on flawed theorems?
• RQ2 What is the finite-sample rate of convergence for Rényi entropy estimation using k-NN graphs under Lipschitz density conditions?
• RQ3 How does the use of generalized k-NN graphs (connecting to more than just the k-th neighbor) affect the estimation accuracy and convergence rate?
• RQ4 Can the k-NN graph framework be extended to mutual information estimation via empirical copulas, and what are the consistency conditions for this extension?
• RQ5 What are the computational advantages of using k-NN graphs over MST and TSP for estimating Rényi entropy and mutual information across multiple α values?

Key findings

• The proposed Rényi entropy estimator is almost surely consistent for α ∈ (0,1) under bounded support, with a rigorous proof correcting prior errors in the literature.
• The paper establishes the first finite-sample high-probability error bounds for Rényi entropy estimation, with rates depending on dimension d and parameter p = d(1−α): O(n^{-(d−p)/(d(2d−p))}) for 0 < p < d−1 and O(n^{-(d−p)/(d(d+1))}) for d−1 ≤ p < d.
• For 0 < p < 1, the error bound scales as O(n^{p/d − p/2}(log(1/δ))^{p/2}), and for 1 ≤ p < d, as O(n^{p/d − 1/2}(log(1/δ))^{1/2}).
• The mutual information estimator based on k-NN graphs and empirical copulas is strongly consistent for d ≥ 3 and α ∈ (1/2, 1), extending prior work that used MST and TSP.
• The k-NN graph framework enables efficient estimation across multiple α values without recomputing the graph, unlike MST and TSP, due to monotonicity of the p-th power transformation.
• Numerical experiments suggest that connecting each point to all k nearest neighbors (generalized k-NN) improves convergence rates compared to connecting only to the k-th neighbor.