[Paper Review] Generalized Exponential Concentration Inequality for Renyi Divergence Estimation

Shashank Singh, Barnabás Póczos|arXiv (Cornell University)|Mar 28, 2016

Sparse and Compressive Sensing TechniquesEngineering25 references9 citations

TL;DR

This paper presents the first finite-sample exponential concentration inequality for a nonparametric Rényi-α divergence estimator using two independent i.i.d. samples on the d-dimensional unit cube. The method employs kernel density estimation with a smooth Hölder-class density assumption and establishes a bound that decays exponentially in sample size, proving the first such concentration result for Rényi divergence estimation with finite samples.

ABSTRACT

Estimating divergences between probability distributions in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main contribution of our work is to provide such a bound for an estimator of Renyi divergence for a smooth Holder class of densities on the d-dimensional unit cube. We also illustrate our theoretical results with a numerical experiment.

Motivation & Objective

• To close the gap in theoretical understanding of Rényi-α divergence estimators by deriving finite-sample convergence guarantees.
• To provide the first exponential concentration inequality for any nonparametric Rényi-α divergence estimator, addressing a long-standing open problem in nonparametric statistics.
• To establish theoretical bounds on bias and variance for a kernel-based Rényi-α divergence estimator under Hölder smoothness conditions.
• To validate the theoretical bounds with a numerical experiment on multivariate normal distributions restricted to [0,1]^d.

Proposed method

• Uses kernel density estimation with a product kernel on the d-dimensional unit cube [0,1]^d to estimate the densities of two underlying distributions from two independent i.i.d. samples.
• Applies Taylor expansion and Hölder continuity assumptions (with smoothness parameter β and norm r) to bound the bias of the density estimator near the boundary.
• Derives a bias bound of order O(h^{β}) for the kernel density estimator, where h is the bandwidth.
• Uses McDiarmid's inequality to derive an exponential concentration bound on the divergence estimator by showing the sensitivity of the divergence to individual sample changes is O(1/n).
• Combines bias and variance bounds using Hölder's inequality and standard kernel density estimation results to derive a total error bound of O(h^{β} + h^{2β} + 1/(n h^d)).
• Optimizes the bandwidth h to achieve a mean squared error of O(n^{-1}) when the densities are infinitely differentiable (β = ∞).

Experimental results

Research questions

• RQ1 Can an exponential concentration inequality be derived for a nonparametric Rényi-α divergence estimator with finite samples?
• RQ2 What is the optimal trade-off between bias and variance in Rényi-α divergence estimation under Hölder smoothness conditions?
• RQ3 How does the convergence rate of the estimator depend on the dimension d and the smoothness β of the underlying densities?
• RQ4 Can the theoretical concentration bound be validated empirically on a realistic statistical model?

Key findings

• The paper establishes the first finite-sample exponential concentration inequality for any nonparametric Rényi-α divergence estimator, proving that the estimator concentrates around the true divergence with high probability at an exponential rate in sample size.
• The bias of the kernel density estimator is bounded by O(h^{β}) for densities in the Hölder class Σκ(β, L, r), where h is the bandwidth and β is the smoothness parameter.
• The variance of the divergence estimator is shown to satisfy an exponential tail bound via McDiarmid's inequality, with the sensitivity of the divergence to any single sample being O(1/n).
• The total mean squared error of the estimator is bounded by O(h^{β} + h^{2β} + 1/(n h^d)), which achieves the optimal rate O(n^{-1}) when the densities are infinitely differentiable (β = ∞).
• Numerical experiments on 3D normal distributions restricted to [0,1]^3 confirm the theoretical convergence rate, showing the empirical mean squared error matches the derived O(n^{-1}) bound.
• The theoretical concentration bound closely tracks the empirical error across sample sizes from 1 to 5000, validating the tightness of the derived inequality.