[Paper Review] On the Expected Size of Conformal Prediction Sets

Guneet S. Dhillon, George Deligiannidis+1 more|arXiv (Cornell University)|Jun 12, 2023

Statistical Methods and Inference4 citations

TL;DR

This paper provides the first finite-sample theoretical analysis of the expected size of prediction sets in split conformal prediction, deriving computable point estimates and high-probability confidence intervals using a single data split. The method enables efficient, one-time estimation of expected set size without repeated Monte Carlo sampling, validated empirically across regression and classification tasks with strong coverage guarantees.

ABSTRACT

While conformal predictors reap the benefits of rigorous statistical guarantees on their error frequency, the size of their corresponding prediction sets is critical to their practical utility. Unfortunately, there is currently a lack of finite-sample analysis and guarantees for their prediction set sizes. To address this shortfall, we theoretically quantify the expected size of the prediction sets under the split conformal prediction framework. As this precise formulation cannot usually be calculated directly, we further derive point estimates and high-probability interval bounds that can be empirically computed, providing a practical method for characterizing the expected set size. We corroborate the efficacy of our results with experiments on real-world datasets for both regression and classification problems.

Motivation & Objective

• To address the lack of finite-sample analysis for the expected size of conformal prediction sets, which is critical for practical deployment.
• To provide theoretically grounded, empirically computable estimates of the expected prediction set size under the split conformal framework.
• To enable efficient, single-run estimation of expected set size for diverse users with varying error tolerance and data constraints.
• To offer high-probability interval bounds that adapt to the non-conformity function and are applicable to both classification and regression.
• To reduce the computational burden of empirical estimation via Monte Carlo averaging by replacing it with a one-time computation using theoretical approximations.

Proposed method

• Derives a theoretical expression for the expected size of split conformal prediction sets under i.i.d. data, based on the distribution of non-conformity scores on calibration data.
• Proposes a point estimate of the expected set size using empirical calibration scores, avoiding repeated model runs.
• Applies the Central Limit Theorem and the Dvoretzky–Kiefer–Wolfowitz inequality to construct high-probability confidence intervals around the expected set size.
• Introduces interval estimation procedures that are adaptive to the non-conformity function and applicable to both classification and regression problems.
• Empirically validates the estimates using synthetic and real-world datasets, comparing point estimates and intervals against Monte Carlo averages.
• Uses a single data split for all estimates, eliminating the need for multiple conformal prediction runs per user configuration.

Experimental results

Research questions

• RQ1 What is the finite-sample expected size of prediction sets in the split conformal prediction framework?
• RQ2 Can we derive a computable point estimate of the expected prediction set size without repeated Monte Carlo sampling?
• RQ3 How can we construct high-probability confidence intervals for the expected prediction set size that are valid across different non-conformity functions?
• RQ4 How do the proposed estimates compare to Monte Carlo averages in terms of accuracy and coverage?
• RQ5 Can the proposed method be applied uniformly across both classification and regression tasks with varying data and error tolerance constraints?

Key findings

• The theoretical expected prediction set size, derived in Theorem 1, is validated as the limit of Monte Carlo averages as the number of calibration points increases.
• The proposed point estimate closely tracks the theoretical expected size and converges to it with increasing calibration data size.
• The confidence intervals constructed via the Central Limit Theorem and the Dvoretzky–Kiefer–Wolfowitz inequality achieve high coverage, with error frequencies below the nominal level (γ=0.1 and γ=0.01) in 99.9% and 100% of cases, respectively.
• The proposed intervals are adaptive to the non-conformity function and applicable to regression, unlike alternative methods such as HI intervals which are limited to classification.
• The method reduces the need for repeated conformal prediction runs, enabling a single data collection and computation to serve multiple users with different error tolerance levels.
• Empirical results on synthetic and real-world datasets confirm that the point estimates and interval bounds are accurate and reliable across diverse settings of $a$, $b$, $m$, and $n$.