[Paper Review] Unbiased Test Error Estimation in the Poisson Means Problem via Coupled Bootstrap Techniques

Natalia L. Oliveira, Lei Jing+1 more|arXiv (Cornell University)|Dec 4, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces a coupled bootstrap (CB) method for unbiased test error estimation in the Poisson means problem, using synthetic binomial noise to generate paired training and test data vectors. The CB estimator is unbiased for the test error when the true mean is shrunk by a factor (1−p), and asymptotically recovers the known unbiased estimator as p→0, without requiring smoothness or structural assumptions on the algorithm.

ABSTRACT

We propose a coupled bootstrap (CB) method for the test error of an arbitrary algorithm that estimates the mean in a Poisson sequence, often called the Poisson means problem. The idea behind our method is to generate two carefully-designed data vectors from the original data vector, by using synthetic binomial noise. One such vector acts as the training sample and the second acts as the test sample. To stabilize the test error estimate, we average this over multiple bootstrap B of the synthetic noise. A key property of the CB estimator is that it is unbiased for the test error in a Poisson problem where the original mean has been shrunken by a small factor, driven by the success probability $p$ in the binomial noise. Further, in the limit as $B o \infty$ and $p o 0$, we show that the CB estimator recovers a known unbiased estimator for test error based on Hudson's lemma, under no assumptions on the given algorithm for estimating the mean (in particular, no smoothness assumptions). Our methodology applies to two central loss functions that can be used to define test error: Poisson deviance and squared loss. Via a bias-variance decomposition, for each loss function, we analyze the effects of the binomial success probability and the number of bootstrap samples and on the accuracy of the estimator. We also investigate our method empirically across a variety of settings, using simulated as well as real data.

Motivation & Objective

• To develop a test error estimator that is unbiased for the Poisson means problem without requiring smoothness or structural assumptions on the estimation algorithm.
• To provide a stable, low-variance alternative to existing unbiased estimators (e.g., based on Hudson’s lemma) that require n+1 algorithm runs.
• To extend the applicability of unbiased estimation to loss functions such as Poisson deviance and squared loss in discrete, count-based models.
• To enable reliable model comparison and tuning by ensuring unbiased estimation of test error differences between algorithms.
• To generalize bootstrap-based error estimation beyond Gaussian models to the Poisson setting via a novel data-splitting mechanism using binomial noise.

Proposed method

• Generate two independent data vectors, $Y^*$ and $Y^lat$, from the original data $Y$ using binomial thinning: $Y^* riangleq \mathrm{Bin}(Y, p)$, $Y^lat \triangleq Y - Y^*$, ensuring $\mathbb{E}[Y^*] = \mathbb{E}[Y^\flat] = (1-p)\mu$.
• Use $Y^*$ as the training sample and $Y^\flat$ as the test sample to compute the algorithm’s prediction $\hat{\mu} = g(Y^*)$, then evaluate loss $L(Y^\flat, g(Y^*))$.
• Repeat the above process $B$ times with independent binomial noise draws and average the loss to form the coupled bootstrap estimator $\widehat{\mathrm{Err}}_{\mathrm{CB}}(g)$.
• Prove that $\mathbb{E}[\widehat{\mathrm{Err}}_{\mathrm{CB}}(g)] = \mathrm{Err}_p(g)$, where $\mathrm{Err}_p(g)$ is the test error of $g$ when the true mean is $\mu_p = (1-p)\mu$, establishing unbiasedness.
• Show that as $p \to 0$, the CB estimator converges to the classical unbiased estimator derived from Hudson’s lemma, which requires $n+1$ algorithm runs.
• Analyze the bias-variance trade-off by decomposing the mean squared error of the estimator in terms of $p$ and $B$, demonstrating improved stability over the standard unbiased estimator when $g$ is unstable.

Experimental results

Research questions

• RQ1 Can a bootstrap-based method be constructed to provide an unbiased estimate of test error in the Poisson means problem without assuming smoothness or structure in the estimation algorithm?
• RQ2 How does the choice of binomial noise parameter $p$ affect the bias and variance of the coupled bootstrap estimator?
• RQ3 Does the coupled bootstrap estimator achieve lower variance than the classical unbiased estimator (based on $n+1$ runs) in practice, especially for unstable algorithms?
• RQ4 To what extent does the coupled bootstrap estimator recover the known unbiased estimator in the limit $p \to 0$?
• RQ5 Can the method be applied effectively to model selection and tuning in Poisson regression and density estimation, particularly when using deviance or squared loss?

Key findings

• The coupled bootstrap estimator is exactly unbiased for the test error of the algorithm when the true mean is shrunk by a factor of $1-p$, regardless of the algorithm $g$.
• As $p \to 0$, the coupled bootstrap estimator converges to the classical unbiased estimator derived from Hudson’s lemma, which is known to be unbiased under no assumptions on $g$.
• The method requires only $B$ runs of the algorithm $g$, compared to $n+1$ runs required by the classical unbiased estimator, offering a significant computational advantage.
• Empirical results show that the coupled bootstrap estimator often has lower variance than the classical unbiased estimator, especially when the algorithm $g$ is unstable.
• In density estimation tasks using P-splines on sloth sighting data, minimizing the CB-estimated deviance loss led to more plausible, smoother density estimates than minimizing CB-estimated squared loss.
• Model selection using the CB estimator successfully identified more regularized tuning parameters when deviance loss was used, aligning better with domain expectations than squared loss.