[Paper Review] Stable Conformal Prediction Sets

Eugène Ndiaye|arXiv (Cornell University)|Dec 19, 2021

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a computationally efficient method for constructing stable conformal prediction sets in regression without data splitting, leveraging algorithmic stability bounds to approximate exact conformal sets with guaranteed finite-sample coverage. The approach requires only a single model fit and achieves tight, reliable confidence intervals when the model is stable and sample size is large.

ABSTRACT

When one observes a sequence of variables $(x_1, y_1), \ldots, (x_n, y_n)$, Conformal Prediction (CP) is a methodology that allows to estimate a confidence set for $y_{n+1}$ given $x_{n+1}$ by merely assuming that the distribution of the data is exchangeable. CP sets have guaranteed coverage for any finite population size $n$. While appealing, the computation of such a set turns out to be infeasible in general, e.g. when the unknown variable $y_{n+1}$ is continuous. The bottleneck is that it is based on a procedure that readjusts a prediction model on data where we replace the unknown target by all its possible values in order to select the most probable one. This requires computing an infinite number of models, which often makes it intractable. In this paper, we combine CP techniques with classical algorithmic stability bounds to derive a prediction set computable with a single model fit. We demonstrate that our proposed confidence set does not lose any coverage guarantees while avoiding the need for data splitting as currently done in the literature. We provide some numerical experiments to illustrate the tightness of our estimation when the sample size is sufficiently large, on both synthetic and real datasets.

Motivation & Objective

• To address the computational infeasibility of exact conformal prediction in regression due to the need to retrain on infinite candidate targets.
• To eliminate the need for data splitting in conformal prediction while preserving finite-sample coverage guarantees.
• To develop a method that leverages algorithmic stability to bound model variation and enable efficient computation of prediction sets.
• To provide a practical, scalable alternative to existing conformal prediction methods that are computationally prohibitive for continuous outcomes.
• To demonstrate that stability-based approximation yields tight confidence sets comparable to exact conformal sets when sample size is sufficient.

Proposed method

• The method replaces the standard data-splitting approach with a stability-based bound on model deviation under input perturbations.
• It uses a single model fit on the full dataset and derives a confidence set based on the stability of the prediction function.
• The core idea is to bound the change in prediction scores when the target label is replaced by a candidate, using Lipschitz continuity and algorithmic stability.
• The prediction set is defined as the set of candidates z for which the score is not too extreme relative to the empirical distribution, adjusted by a stability term.
• The method constructs a confidence set via a stability-corrected p-value function that avoids retraining for each candidate z.
• It relies on a theoretical bound on the difference between prediction scores under perturbed labels, derived from the stability of the learning algorithm.

Experimental results

Research questions

• RQ1 Can we construct conformal prediction sets for regression without data splitting while preserving finite-sample coverage?
• RQ2 How can algorithmic stability be used to approximate conformal prediction sets efficiently?
• RQ3 What is the trade-off between computational efficiency and accuracy in conformal prediction when avoiding data splitting?
• RQ4 How does the tightness of the confidence set depend on model stability and sample size?
• RQ5 Can stability bounds be used to derive a single-fit method that matches the performance of exact conformal prediction?

Key findings

• The proposed method achieves exact finite-sample coverage without data splitting, maintaining the theoretical guarantees of conformal prediction.
• The method is computationally efficient, being 20–30 times faster than standard split conformal methods in experiments.
• On synthetic and real datasets, the stable conformal set (stabCP) closely approximates the exact conformal set (rootCP), especially as sample size increases.
• The method performs best when the model is stable and the sample size is large, with stability bounds decaying as O(1/n).
• For small sample sizes, the method loses precision due to unreliable stability estimates, but coverage remains valid.
• The approach is not applicable when stability bounds are unknown or poorly estimated, particularly for non-convex models like deep neural networks.