[Paper Review] Universal distribution of the empirical coverage in split conformal prediction

Paulo C. Marques F.|arXiv (Cornell University)|Mar 5, 2023

Statistical Methods and Inference4 citations

TL;DR

This paper establishes the exact finite-sample and asymptotic distributions of the empirical coverage in split conformal prediction under exchangeability. It shows that for any nominal miscoverage level α and calibration sample size n, the coverage distribution is universally distributed as a Beta(⌈(1−α)(n+1)⌉, ⌊α(n+1)⌋) distribution, both for finite horizons and in the almost sure limit as the number of future predictions grows.

ABSTRACT

When split conformal prediction operates in batch mode with exchangeable data, we determine the exact distribution of the empirical coverage of prediction sets produced for a finite batch of future observables, as well as the exact distribution of its almost sure limit when the batch size goes to infinity. Both distributions are universal, being determined solely by the nominal miscoverage level and the calibration sample size, thereby establishing a criterion for choosing the minimum required calibration sample size in applications.

Motivation & Objective

• To characterize the exact finite-sample distribution of the empirical coverage in split conformal prediction under exchangeable data.
• To derive the almost sure limit distribution of the empirical coverage as the number of future predictions tends to infinity.
• To establish that both distributions are universal, depending only on the nominal miscoverage level α and calibration sample size n.
• To provide a theoretical foundation for understanding coverage variability in practical split conformal prediction applications.
• To support practical calibration sample size selection by quantifying the distributional behavior of coverage under finite and asymptotic conditions.

Proposed method

• Leverages data exchangeability to establish exchangeability of conformity scores and coverage indicators.
• Applies de Finetti’s representation theorem to model the almost sure limit of coverage as a Beta-distributed random variable.
• Uses Pólya’s urn scheme as a probabilistic analogy to derive the exact finite-horizon distribution of empirical coverage.
• Derives the exact distribution of the future coverage $ C^{(n,eta)}_m $ as a function of $ m $, $ n $, and $ \alpha $, using combinatorial integration over the de Finetti mixing measure.
• Characterizes the mixing measure $ \mu_\Theta $ as a Beta distribution with parameters $ b = \lceil(1-\alpha)(n+1)\rceil $, $ g = \lfloor\alpha(n+1)\rfloor $, leading to the exact coverage law.
• Applies Scheffé’s theorem to establish asymptotic normality of the coverage deviation from the nominal level.

Figure 1: Simulation of the future coverage discussed in Example 5 .

Experimental results

Research questions

• RQ1 What is the exact finite-sample distribution of the empirical coverage in split conformal prediction under exchangeability?
• RQ2 What is the almost sure limit distribution of the empirical coverage as the number of future predictions increases?
• RQ3 How does the distribution of the empirical coverage depend universally on the nominal miscoverage level and calibration sample size?
• RQ4 Can the coverage distribution be characterized using de Finetti’s theorem and Pólya’s urn scheme in this framework?
• RQ5 What sample size is required to ensure high-probability concentration of the coverage around the nominal level?

Key findings

• The empirical coverage $ C^{(n,\alpha)}_m $ for a finite horizon $ m $ has an exact distribution that depends only on $ \alpha $ and $ n $, derived via exchangeability and combinatorial integration.
• The almost sure limit $ C^{(n,\alpha)}_\infty $ of the empirical coverage is distributed as $ \mathrm{Beta}(\lceil(1-\alpha)(n+1)\rceil, \lfloor\alpha(n+1)\rfloor) $, a universal result under exchangeability.
• The expected value of the limiting coverage is $ \mathbb{E}[C^{(n,\alpha)}_\infty] = 1 - \alpha + O(1/n) $, showing asymptotic consistency with the nominal level.
• The variance of the limiting coverage is $ \mathrm{Var}[C^{(n,\alpha)}_\infty] = \frac{\alpha(1-\alpha)}{n} + O(1/n^2) $, indicating convergence at rate $ 1/\sqrt{n} $.
• The normalized deviation $ \sqrt{n}(C^{(n,\alpha)}_\infty - (1-\alpha)) $ converges in distribution to a normal distribution with mean 0 and variance $ \alpha(1-\alpha) $.
• For practical calibration, a sample size of $ n = 860 $ ensures that the limiting coverage lies within $ \pm 0.02 $ of $ 1 - \alpha = 0.9 $ with at least 95% probability.

Figure 2: A connection between split conformal prediction and Pólya’s urn scheme.