[Paper Review] Bootstrap prediction intervals with asymptotic conditional validity and unconditional guarantees

Yunyi Zhang, Dimitris N. Politis|arXiv (Cornell University)|May 19, 2020

Statistical Methods and Inference27 references4 citations

TL;DR

This paper proposes a novel bootstrap algorithm for constructing prediction intervals in linear models that achieve asymptotically valid conditional coverage while guaranteeing an unconditional lower bound on coverage probability. By reweighting bootstrap residuals and calibrating quantiles using asymptotic theory, the method ensures that the conditional coverage probability converges in probability to the nominal level, and the risk of conditional under-coverage is asymptotically controlled.

ABSTRACT

It can be argued that optimal prediction should take into account all available data. Therefore, to evaluate a prediction interval's performance one should employ conditional coverage probability, conditioning on all available observations. Focusing on a linear model, we derive the asymptotic distribution of the difference between the conditional coverage probability of a nominal prediction interval and the conditional coverage probability of a prediction interval obtained via a residual-based bootstrap. Applying this result, we show that a prediction interval generated by the residual-based bootstrap has approximately 50% probability to yield conditional under-coverage. We then develop a new bootstrap algorithm that generates a prediction interval that asymptotically controls both the conditional coverage probability as well as the possibility of conditional under-coverage. We complement the asymptotic results with several finite-sample simulations.

Motivation & Objective

• To address the limitation of standard residual-based bootstrap prediction intervals, which may suffer from conditional under-coverage despite asymptotic validity.
• To develop a prediction interval that simultaneously achieves asymptotic conditional coverage probability close to the nominal level and provides an unconditional lower bound on coverage.
• To formalize the performance of prediction intervals in terms of conditional coverage, which better reflects real-world prediction practice where all data are observed.
• To bridge the gap between asymptotic conditional validity and finite-sample unconditional guarantees in prediction interval construction.

Proposed method

• The method introduces a reweighted residual-based bootstrap procedure that adjusts the empirical distribution of residuals to improve conditional coverage properties.
• It derives the asymptotic distribution of the difference between the true conditional coverage probability and the bootstrap-based coverage, enabling calibration of the bootstrap quantiles.
• The algorithm constructs prediction intervals using a modified quantile from the bootstrap distribution, where the quantile is selected to ensure both conditional validity and a lower bound on coverage.
• The approach uses a double-bootstrap framework: one level for estimating the conditional distribution of prediction errors, and a second level to control for potential under-coverage.
• Key components include the use of studentized residuals, empirical process theory, and concentration inequalities to bound the deviation between bootstrap and true conditional coverage.
• The method is calibrated using asymptotic expansions involving the empirical process of residuals and the limiting normal distribution of studentized residuals.

Figure 1: Point-wise prediction intervals for the linear model $y_{i}=0.8+0.5x_{i}+\epsilon_{i},i=1,2,...,100$ . Black line, red dashed lines, solid purple lines, blue dashed lines, solid green lines respectively plot predictors, and point-wise prediction intervals generated by residual-based bootst

Experimental results

Research questions

• RQ1 Can a bootstrap-based prediction interval be constructed such that its conditional coverage probability converges in probability to the nominal level (1−α)?
• RQ2 What is the asymptotic behavior of the conditional under-coverage probability for standard residual-based bootstrap intervals?
• RQ3 Can a prediction interval be designed to simultaneously achieve asymptotic conditional validity and an unconditional lower bound on coverage probability?
• RQ4 How can the bootstrap procedure be modified to control the risk of conditional under-coverage while maintaining asymptotic validity?
• RQ5 What theoretical tools are required to establish both conditional and unconditional coverage guarantees in the context of linear models?

Key findings

• The standard residual-based bootstrap yields a prediction interval with approximately 50% probability of conditional under-coverage, indicating a significant risk of underperformance in finite samples.
• The proposed bootstrap algorithm asymptotically controls both the conditional coverage probability and the risk of conditional under-coverage, ensuring that the conditional coverage probability converges in probability to the nominal level.
• The method guarantees an unconditional lower bound on coverage probability, satisfying the requirement of conformal prediction while improving upon it with conditional validity.
• Finite-sample simulations confirm that the proposed method maintains coverage close to the nominal level across various design configurations and error distributions.
• Theoretical analysis shows that the deviation between the bootstrap-based conditional coverage and the nominal level converges to zero in probability, with convergence rates controlled by the empirical process of residuals.
• The method achieves its goals by reweighting bootstrap residuals and using a refined quantile calibration that accounts for the variability in the estimated error variance.

(a) Normal, Residual-based bootstrap