[Paper Review] Conformal prediction beyond exchangeability

Rina Foygel Barber, Emmanuel J. Candès+2 more|arXiv (Cornell University)|Feb 27, 2022

Energy Load and Power Forecasting7 citations

TL;DR

This paper generalizes conformal prediction to handle non-exchangeable data and non-symmetric algorithms by introducing weighted quantiles and a novel randomization technique, enabling provably robust coverage even under distribution drift or asymmetric model fitting, with coverage loss bounded by a weighted sum of total variation distances between data permutations.

ABSTRACT

Conformal prediction is a popular, modern technique for providing valid predictive inference for arbitrary machine learning models. Its validity relies on the assumptions of exchangeability of the data, and symmetry of the given model fitting algorithm as a function of the data. However, exchangeability is often violated when predictive models are deployed in practice. For example, if the data distribution drifts over time, then the data points are no longer exchangeable; moreover, in such settings, we might want to use a nonsymmetric algorithm that treats recent observations as more relevant. This paper generalizes conformal prediction to deal with both aspects: we employ weighted quantiles to introduce robustness against distribution drift, and design a new randomization technique to allow for algorithms that do not treat data points symmetrically. Our new methods are provably robust, with substantially less loss of coverage when exchangeability is violated due to distribution drift or other challenging features of real data, while also achieving the same coverage guarantees as existing conformal prediction methods if the data points are in fact exchangeable. We demonstrate the practical utility of these new tools with simulations and real-data experiments on electricity and election forecasting.

Motivation & Objective

• Address the limitation of conformal prediction under distribution drift or non-i.i.d. data, where exchangeability assumptions fail.
• Enable the use of non-symmetric algorithms that prioritize recent or relevant data points in time-series or evolving data settings.
• Provide theoretical coverage guarantees that remain close to nominal levels even when exchangeability is violated.
• Develop a framework that maintains full coverage validity under minimal assumptions on data distribution, including no i.i.d. or exchangeability requirements.
• Introduce a weighted residual-based approach to quantify and bound the loss in coverage due to non-exchangeability.

Proposed method

• Introduce weighted quantiles in the conformal prediction framework to downweight distant or less relevant data points during calibration.
• Propose a randomization technique that allows non-symmetric algorithms to be used while preserving valid coverage guarantees.
• Define a coverage gap as the difference between nominal coverage and actual coverage, bounded by a weighted sum of total variation distances between data vectors with and without the test point.
• Use a weighted empirical distribution of residuals to define 'strange' points, where the weighted fraction of such points is bounded by α.
• Establish a probabilistic bound on non-coverage by relating it to the event that the test point is classified as 'strange' under a permuted data configuration.
• Leverage the total variation distance between residual vectors under data permutations to quantify sensitivity to non-exchangeability, ensuring robustness.

Experimental results

Research questions

• RQ1 How can conformal prediction be extended to maintain valid coverage when data points are not exchangeable due to distribution drift?
• RQ2 What modifications to the conformal prediction framework allow for the use of non-symmetric algorithms that treat recent or relevant data more heavily?
• RQ3 Can a theoretical bound on coverage loss be derived under non-exchangeable data, and what does it depend on?
• RQ4 How does the proposed method compare to standard conformal prediction in terms of coverage accuracy when exchangeability is violated?
• RQ5 What role do data weights and randomization play in stabilizing coverage under non-i.i.d. or non-exchangeable data?

Key findings

• The coverage gap—defined as the difference between nominal and actual coverage—is bounded by a weighted sum of total variation distances between the residual vectors obtained by swapping the test point with each training point.
• When data are exchangeable, the proposed method achieves the same coverage guarantees as standard conformal prediction, ensuring consistency with existing theory.
• In the presence of distribution drift, the method maintains coverage close to the nominal level, as demonstrated in simulations and real-data experiments on electricity and election forecasting.
• The use of weighted quantiles and randomization enables robustness against non-exchangeable data without requiring strong distributional assumptions.
• The theoretical bound on coverage loss depends only on the total variation distances between data permutations, making it applicable even when the joint distribution of data is unknown.
• Empirical results on the ELEC2 dataset show that standard conformal prediction loses coverage over time due to drift, while the proposed nonexchangeable method maintains approximately 90% coverage.