[Paper Review] Dualizing Le Cam's method for functional estimation, with applications to estimating the unseens

Yury Polyanskiy, Yihong Wu|arXiv (Cornell University)|Feb 14, 2019

Statistical Methods and Inference50 references10 citations

TL;DR

This paper introduces a dualized formulation of Le Cam's two-point method for functional estimation, revealing that the minimax risk in estimating linear functionals is tightly characterized by a bias-variance tradeoff through convex duality. It establishes tight lower bounds for estimating unseen quantities—such as distinct elements and unseen species—under various models, including exponential families and high-dimensional settings, with explicit phase transitions (elbow effects) at critical sampling thresholds.

ABSTRACT

Le Cam's method (or the two-point method) is a commonly used tool for obtaining statistical lower bound and especially popular for functional estimation problems. This work aims to explain and give conditions for the tightness of Le Cam's lower bound in functional estimation from the perspective of convex duality. Under a variety of settings it is shown that the maximization problem that searches for the best two-point lower bound, upon dualizing, becomes a minimization problem that optimizes the bias-variance tradeoff among a family of estimators. For estimating linear functionals of a distribution our work strengthens prior results of Donoho-Liu \cite{DL91} (for quadratic loss) by dropping the Hölderian assumption on the modulus of continuity. For exponential families our results extend those of Juditsky-Nemirovski \cite{JN09} by characterizing the minimax risk for the quadratic loss under weaker assumptions on the exponential family. We also provide an extension to the high-dimensional setting for estimating separable functionals. Notably, coupled with tools from complex analysis, this method is particularly effective for characterizing the ``elbow effect'' -- the phase transition from parametric to nonparametric rates. As the main application we derive sharp minimax rates in the Distinct elements problem (given a fraction $p$ of colored balls from an urn containing $d$ balls, the optimal error of estimating the number of distinct colors is $ ilde Θ(d^{-\frac{1}{2}\min\{\frac{p}{1-p},1\}})$) and the Fisher's species problem (given $n$ iid observations from an unknown distribution, the optimal prediction error of the number of unseen symbols in the next (unobserved) $r \cdot n$ observations is $ ilde Θ(n^{-\min\{\frac{1}{r+1},\frac{1}{2}\}})$).

Motivation & Objective

• To explain the tightness of Le Cam's lower bound in functional estimation via convex duality.
• To extend prior results on minimax risk for linear functionals by removing Hölderian assumptions on the modulus of continuity.
• To characterize minimax risk in exponential families under weaker assumptions than previous work.
• To develop a unified framework for high-dimensional separable functional estimation with explicit phase transitions.
• To apply the method to three key problems in 'estimating the unseens'—distinct elements, Fisher’s species problem, and population recovery—recovering and extending known results.

Proposed method

• Dualize the two-point Le Cam lower bound optimization problem into a minimization over estimators, revealing a bias-variance tradeoff structure.
• Use convex duality to transform the maximization over distributions into a minimization over estimators, enabling tighter risk characterization.
• Apply tools from complex analysis and approximation theory to analyze the $χ^2$-divergence and moment-matching constraints.
• Characterize the minimax risk via the $χ^2$-divergence in i.i.d. and deterministic settings, with a duality equivalence up to constant factors.
• Employ Poissonization and de-Poissonization techniques to relate fixed-sample and Poisson-sampled models in the species problem.
• Leverage Hermite polynomials and moment-matching arguments to construct feasible distributions achieving the lower bound.

Experimental results

Research questions

• RQ1 Under what conditions is Le Cam’s two-point lower bound tight for functional estimation?
• RQ2 How can convex duality be used to reframe the minimax risk as a bias-variance optimization problem?
• RQ3 What is the minimax rate for estimating the number of distinct colors in a population when only a fraction of items are observed?
• RQ4 What is the optimal prediction error for estimating the number of unseen symbols in future samples, and where does the phase transition (elbow) occur?
• RQ5 How does the method extend to high-dimensional or separable functionals in exponential families?

Key findings

• The minimax risk for estimating linear functionals is tightly characterized by the dualized Le Cam lower bound, with $ R_{\text{iid}}^{*}(n) \asymp R_{\text{det}}^{*}(n) \asymp \max_{\theta,\theta'} \left\{ |T(\pi) - T(\pi')|^2 : \chi^2(\pi P \| \pi' P) \leq \frac{1}{n} \right\} $, up to universal constants.
• For the distinct elements problem, the normalized error is within logarithmic factors of $ d^{-\frac{1}{2} \min\{\frac{p}{1-p}, 1\}} $, with an elbow at $ p = \frac{1}{2} $.
• For Fisher’s species problem, the normalized prediction error is within logarithmic factors of $ n^{-\min\{\frac{1}{r+1}, \frac{1}{2}\}} $, exhibiting an elbow at $ r = 1 $.
• The method recovers the prior result of [PSW17] on population recovery and extends it to new settings with explicit phase transitions.
• The analysis via Hermite polynomials and moment matching shows that the minimax risk for $ \mathbb{E}[|\theta|] $ under $ \theta \in [-1,1] $ is bounded by $ \tilde{O}(t^{1/2}) $, with matching lower bounds via approximation theory.
• The Poissonized and fixed-sample models are shown to be within $ O(\log n / n) $ of each other, enabling robust risk analysis across models.