[Paper Review] A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Anastasios N. Angelopoulos, Stephen Bates|arXiv (Cornell University)|Jul 15, 2021

Anomaly Detection Techniques and Applications11 references28 citations

TL;DR

This paper presents conformal prediction as a distribution-free framework for producing valid uncertainty sets for predictions across classification and regression, with practical procedures, extensions, and diagnostics.

ABSTRACT

Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.

Motivation & Objective

• Provide a practical, self-contained introduction to conformal prediction and distribution-free uncertainty quantification.
• Show how conformal prediction converts heuristic model uncertainty into rigorous, finite-sample guarantees.
• Demonstrate multiple conformal procedures for classification and regression with real-data examples.
• Discuss evaluation, adaptivity, and extensions to complex tasks and distribution shifts.

Proposed method

• Introduce the conformal prediction framework with a calibration step using a score function and a quantile qhat to form prediction sets.
• Present split conformal prediction as the primary, widely-used variant with formal coverage guarantees.
• Provide concrete score-function-based procedures for classification (adaptive prediction sets), conformalized quantile regression for regression, conformalizing scalar uncertainty estimates, and conformalizing Bayes.
• Explain how the coverage guarantee is independent of the underlying model or data distribution.
• Discuss extensions to structured outputs, distribution shift, outliers, and selective decisions with practical Python code snippets and notebooks.
• Outline evaluation metrics for adaptivity and correctness, including coverage checks and calibration set size considerations.

Experimental results

Research questions

• RQ1 How can one obtain distribution-free, finite-sample uncertainty guarantees for predictions from any model?
• RQ2 How should one design score functions to produce useful, adaptive prediction sets in classification and regression?
• RQ3 What are practical extensions of conformal prediction to tasks like time series, outliers, and distribution drift?
• RQ4 How can one evaluate conformal predictors for adaptivity and correct coverage in real data?
• RQ5 What is the impact of calibration set size on the guarantees and performance?

Key findings

• Conformal prediction yields prediction sets with marginal coverage guaranteed at 1−α, regardless of the model or data distribution.
• Adaptive prediction sets can be constructed to balance set size with input difficulty, improving practical usefulness.
• Conformalized quantile regression provides valid intervals for regression by adjusting existing quantile models with a calibration-derived quantile.
• Various conformal extensions (e.g., to Bayes, scalar uncertainties, and Bayes-optimal sets) are feasible and retain distribution-free guarantees.
• Calibration set size and diagnostic tools are crucial for ensuring correct coverage and measuring adaptivity in practice.