[Paper Review] Adaptive Conformal Inference Under Distribution Shift

Isaac Gibbs, Emmanuel J. Candès|arXiv (Cornell University)|Jun 1, 2021

Machine Learning and Data Classification27 references27 citations

TL;DR

This paper introduces Adaptive Conformal Inference (ACI), an online framework that maintains target coverage for prediction sets under non-stationary data by online updating a single shift parameter. It integrates with any black-box predictor and calibrates predictions to handle distribution drift.

ABSTRACT

We develop methods for forming prediction sets in an online setting where the data generating distribution is allowed to vary over time in an unknown fashion. Our framework builds on ideas from conformal inference to provide a general wrapper that can be combined with any black box method that produces point predictions of the unseen label or estimated quantiles of its distribution. While previous conformal inference methods rely on the assumption that the data points are exchangeable, our adaptive approach provably achieves the desired coverage frequency over long-time intervals irrespective of the true data generating process. We accomplish this by modelling the distribution shift as a learning problem in a single parameter whose optimal value is varying over time and must be continuously re-estimated. We test our method, adaptive conformal inference, on two real world datasets and find that its predictions are robust to visible and significant distribution shifts.

Motivation & Objective

• Motivate and develop prediction-set procedures with guaranteed coverage in an online setting where the data distribution drifts over time.
• Provide a simple, model-agnostic calibration mechanism that maintains long-run coverage without assuming stationarity.
• Demonstrate robustness of the method on real-world datasets subject to distribution shifts.

Proposed method

• Use a conformity-score framework that can be paired with any predictor output (point predictions or quantiles).
• Introduce an online update rule for a shift-adjusted alpha parameter: alpha_{t+1} = alpha_t + gamma (alpha - err_t).
• Define miscoverage err_t based on whether Y_t falls outside the adaptive prediction set with alpha_t.
• Allow calibration of alpha_t to achieve approximate or exact marginal coverage M_t(alpha_t) ≈ alpha through adaptive re-estimation of alpha_t.
• Discuss two simple update variants (equations (2) and (3)) and discuss tradeoffs in step-size gamma.
• Provide theoretical guarantees under distribution shift, including long-run coverage and concentration bounds in a hidden Markov model setting.

Experimental results

Research questions

• RQ1 Can conformal prediction sets be maintained with valid coverage when the data distribution shifts over time?
• RQ2 How can a single-parameter online adaptation detect and compensate for distribution drift while remaining agnostic to the data-generating process?
• RQ3 What are the theoretical guarantees (long-run coverage, concentration) for adaptive conformal inference under non-stationarity?
• RQ4 How does ACI perform on real-world datasets with visible distribution shifts (e.g., financial volatility, election night predictions)?

Key findings

• ACI achieves the target long-run coverage frequency irrespective of the data-generating process.
• With stable shifts, ACI can achieve approximate marginal coverage at most time steps.
• The step-size gamma trades off adaptivity and stability; moderate values (e.g., gamma = 0.005) yielded stable yet responsive behavior in experiments.
• ACI remains robust to significant distribution shifts, performing well where non-adaptive conformal methods fail (e.g., during financial crises in volatility prediction).
• The method can be combined with conformal quantile regression and other conformity scores to preserve coverage while enabling shorter prediction sets when models are accurate.