[Paper Review] Estimation of R\\'enyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

Dávid Pál, Barnabás Póczos+1 more|arXiv (Cornell University)|Mar 9, 2010

Blind Source Separation Techniques39 references73 citations

TL;DR

This paper proposes a nonparametric estimator for Rényi entropy and mutual information using generalized nearest-neighbor graphs and empirical copula transformations. It establishes almost sure consistency and provides finite-sample high-probability error bounds under Lipschitz density assumptions, marking the first rate-of-convergence analysis for Rényi entropy estimation with this class of estimators.

ABSTRACT

We present simple and computationally efficient nonparametric estimators of R\\'enyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\\R^d$. The estimators are calculated as the sum of $p$-th powers of the Euclidean lengths of the edges of the `generalized nearest-neighbor' graph of the sample and the empirical copula of the sample respectively. For the first time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis.

Motivation & Objective

• To develop a computationally efficient, nonparametric estimator of Rényi entropy and mutual information without relying on density estimation.
• To correct and rigorously prove the almost sure consistency of k-NN graph-based Rényi entropy estimators, addressing flaws in prior proofs.
• To establish the first finite-sample high-probability error bounds (rates of convergence) for Rényi entropy estimation under Lipschitz density conditions.
• To extend the method to Rényi mutual information estimation using empirical copulas and k-NN graphs, proving strong consistency for d ≥ 3 and α ∈ (1/2, 1).
• To demonstrate the practical advantage of using generalized k-NN graphs (all k nearest neighbors) over MST or TSP for multiple α values due to reusability of the graph structure.

Proposed method

• Estimates Rényi entropy as the sum of the p-th powers of the Euclidean lengths of edges in a generalized k-NN graph, where each point connects to an arbitrary subset of its k nearest neighbors.
• Uses the empirical copula transformation to map the original i.i.d. sample to the unit cube [0,1]^d, enabling copula-based mutual information estimation.
• Applies a perturbation analysis to bound the difference between the true copula-based graph length and the empirical copula-based length under i.i.d. sampling.
• Employs a novel proof technique based on metric entropy and covering arguments to establish consistency and convergence rates, avoiding flawed applications of Helly-Bray or Fatou-type theorems.
• Derives high-probability error bounds using concentration inequalities and the Lipschitz continuity of the density, leading to explicit convergence rates in terms of sample size n and dimension d.
• Uses the fact that p-th power of distances is monotonic, so the k-NN graph structure remains invariant across different α values (via p = d(1−α)), enabling efficient multi-α estimation.

Experimental results

Research questions

• RQ1 Can a nonparametric Rényi entropy estimator based on generalized k-NN graphs be proven almost surely consistent, and if so, under what conditions?
• RQ2 What is the finite-sample rate of convergence for Rényi entropy estimation using k-NN graphs, assuming the underlying density is Lipschitz continuous?
• RQ3 Can the k-NN graph-based approach be extended to Rényi mutual information estimation via empirical copulas, and is it strongly consistent?
• RQ4 How does the use of generalized k-NN graphs (all k neighbors) compare to previous methods like MST or TSP in terms of computational efficiency and reusability across different α values?
• RQ5 What are the theoretical error bounds for the empirical copula transformation when estimating Rényi mutual information using k-NN graphs?

Key findings

• The proposed Rényi entropy estimator based on generalized k-NN graphs is almost surely consistent for α ∈ (0,1) and d ≥ 1, under bounded support of the density.
• For Lipschitz continuous densities, the paper establishes the first finite-sample high-probability error bounds for Rényi entropy estimation, with rates depending on dimension d and parameter p = d(1−α).
• The error bound for Rényi entropy scales as O(n^{−(d−p)/(d(2d−p))}) for 0 < p < d−1 and O(n^{−(d−p)/(d(d+1))}) for d−1 ≤ p < d, with logarithmic corrections.
• The mutual information estimator based on empirical copulas and k-NN graphs is strongly consistent for d ≥ 3 and α ∈ (1/2,1), extending prior work that used MST or TSP.
• The k-NN graph structure is invariant under monotonic transformations of distance, enabling efficient estimation across multiple α values without recomputing the graph.
• Numerical experiments suggest that using all k nearest neighbors (generalized k-NN) improves the convergence rate compared to connecting only to the k-th neighbor.