[Paper Review] Predictive Inference Is Free with the Jackknife+-after-Bootstrap

Byol Kim, Xu Chen+1 more|arXiv (Cornell University)|Feb 20, 2020

Machine Learning and Data Classification37 references32 citations

TL;DR

Introduces the jackknife+-after-bootstrap (J+aB), a computationally efficient wrapper for ensemble predictors that yields distribution-free predictive intervals with a guaranteed coverage of at least 1−2α, while incurring cost close to a single ensemble prediction.

ABSTRACT

Ensemble learning is widely used in applications to make predictions in complex decision problems---for example, averaging models fitted to a sequence of samples bootstrapped from the available training data. While such methods offer more accurate, stable, and robust predictions and model estimates, much less is known about how to perform valid, assumption-lean inference on the output of these types of procedures. In this paper, we propose the jackknife+-after-bootstrap (J+aB), a procedure for constructing a predictive interval, which uses only the available bootstrapped samples and their corresponding fitted models, and is therefore "free" in terms of the cost of model fitting. The J+aB offers a predictive coverage guarantee that holds with no assumptions on the distribution of the data, the nature of the fitted model, or the way in which the ensemble of models are aggregated---at worst, the failure rate of the predictive interval is inflated by a factor of 2. Our numerical experiments verify the coverage and accuracy of the resulting predictive intervals on real data.

Motivation & Objective

• Motivate valid, assumption-lean predictive inference for ensemble learning outputs.
• Wrap an ensemble method with a predictive interval that requires no distributional assumptions.
• Preserve computational efficiency by reusing base-model fits through out-of-bag observations.
• Provide finite-sample coverage guarantees and empirical verification on real data.

Proposed method

• Propose jackknife+-after-bootstrap (J+aB) as a wrapper around any base regression algorithm and aggregation function.
• Reuse out-of-bag models to obtain leave-one-out predictions without extra base-model calls (Algorithm 2).
• Compute prediction intervals using a joint quantile-based scheme: at each x, C_alpha,n,B(x) = [q_{alpha,n}^{-} { mu_phi extbackslash i (x) - R_i }, q_{alpha,n}^{+} { mu_phi extbackslash i (x) + R_i }], where R_i = |Y_i - mu_phi extbackslash i (X_i)|.
• Ensure computational cost is O(B) base-model calls, matching the cost of a single ensemble prediction.
• Provide a symmetry-corrected version with Binomial(B) draws to restore exchangeability and enable distribution-free guarantees.
• Theoretical guarantee: under IID data and symmetry assumptions, P(Y_{n+1} ∈ C_alpha,n,B^{J+aB}(X_{n+1})) ≥ 1−2α (finite-sample, non-asymptotic).

Experimental results

Research questions

• RQ1 Can a wrapper around ensemble predictors yield valid, distribution-free predictive intervals without strong modeling assumptions?
• RQ2 What is the coverage guarantee and its tightness for J+aB in finite samples?
• RQ3 Can the method maintain computational efficiency comparable to a single ensemble pass while providing consumer-friendly predictive intervals?
• RQ4 How does J+aB perform empirically across real datasets with different base learners?

Key findings

• The J+aB interval achieves coverage close to the nominal level 1−α in experiments on three real datasets.
• Theoretical guarantee shows a distribution-free lower bound of 1−2α for coverage in the worst case, non-asymptotic and valid for any n and distribution under the stated assumptions.
• Computational cost of J+aB is on par with producing a single ensemble prediction, making it effectively “free” in terms of additional model fitting.
• Empirical results indicate that J+aB intervals are informative and often narrower for unstable base learners (e.g., Random Forest) compared to stable ones (e.g., Ridge).
• J+aB compares favorably with the alternative J+ensemble in terms of cost, while delivering comparable coverage and interval quality.
• The method is compatible with various conformity measures (residuals, quantile regression, weighted residuals) and different aggregations.