[Paper Review] Efficient two-sample functional estimation and the super-oracle phenomenon

Thomas B. Berrett, Richard J. Samworth|arXiv (Cornell University)|Apr 18, 2019

Statistical Methods and Inference28 references10 citations

TL;DR

This paper proposes a weighted nearest neighbor estimator for two-sample integral functionals, such as divergences between unknown probability densities. It establishes asymptotic efficiency and a central limit theorem, showing that the estimator can outperform the oracle estimator in worst-case scenarios—a phenomenon termed the 'super-oracle' effect.

ABSTRACT

We consider the estimation of two-sample integral functionals, of the type that occur naturally, for example, when the object of interest is a divergence between unknown probability densities. Our first main result is that, in wide generality, a weighted nearest neighbour estimator is efficient, in the sense of achieving the local asymptotic minimax lower bound. Moreover, we also prove a corresponding central limit theorem, which facilitates the construction of asymptotically valid confidence intervals for the functional, having asymptotically minimal width. One interesting consequence of our results is the discovery that, for certain functionals, the worst-case performance of our estimator may improve on that of the natural `oracle' estimator, which is given access to the values of the unknown densities at the observations.

Motivation & Objective

• To develop a statistically efficient estimator for two-sample integral functionals, such as those arising in density divergence estimation.
• To establish theoretical guarantees—specifically, asymptotic efficiency and a central limit theorem—for the proposed estimator.
• To investigate whether the estimator's worst-case performance can surpass that of the oracle estimator with access to true density values.
• To enable the construction of confidence intervals with asymptotically minimal width for the functional of interest.

Proposed method

• Proposes a weighted nearest neighbor estimator tailored for two-sample integral functionals.
• Derives conditions under which the estimator achieves the local asymptotic minimax lower bound, establishing asymptotic efficiency.
• Establishes a central limit theorem for the estimator, enabling asymptotically valid confidence intervals.
• Analyzes the worst-case risk of the estimator and compares it to the oracle estimator's performance.
• Uses nonparametric techniques and asymptotic theory to derive finite-sample approximations and limiting distributions.
• Applies the estimator to functionals involving divergences between unknown densities, such as Kullback-Leibler or f-divergences.

Experimental results

Research questions

• RQ1 Can a weighted nearest neighbor estimator achieve asymptotic efficiency for two-sample integral functionals?
• RQ2 Does the proposed estimator satisfy a central limit theorem, enabling valid inference via confidence intervals?
• RQ3 Under what conditions can the estimator's worst-case performance exceed that of the oracle estimator?
• RQ4 How does the estimator's performance compare to the oracle when the true densities are known at the observation points?
• RQ5 What is the theoretical justification for the observed 'super-oracle' phenomenon in this estimation framework?

Key findings

• The proposed weighted nearest neighbor estimator is asymptotically efficient, achieving the local asymptotic minimax lower bound for a broad class of two-sample integral functionals.
• A central limit theorem is established for the estimator, allowing the construction of asymptotically valid confidence intervals with asymptotically minimal width.
• The estimator exhibits a 'super-oracle' phenomenon, where its worst-case risk can be strictly smaller than that of the oracle estimator with access to true density values.
• This super-oracle behavior occurs for certain functionals, indicating that nonparametric estimation can outperform parametric oracle benchmarks in worst-case scenarios.
• The results hold under general regularity conditions, making the method widely applicable to divergences and other integral functionals.
• The theoretical framework supports practical inference, including interval estimation, with optimal asymptotic properties.