[Paper Review] Conformal prediction set for time-series

Xu Chen, Yao Xie|arXiv (Cornell University)|Jun 15, 2022

Neural Networks and Applications4 citations

TL;DR

This paper proposes ERAPS, an ensemble-based, distribution-free conformal prediction framework for time-series classification that constructs valid prediction sets without assuming data exchangeability. It theoretically bounds coverage gaps and empirically achieves smaller, more accurate prediction sets than competing methods like SAPS and Naive, maintaining valid marginal and conditional coverage even under complex temporal dependencies.

ABSTRACT

When building either prediction intervals for regression (with real-valued response) or prediction sets for classification (with categorical responses), uncertainty quantification is essential to studying complex machine learning methods. In this paper, we develop Ensemble Regularized Adaptive Prediction Set (ERAPS) to construct prediction sets for time-series (with categorical responses), based on the prior work of [Xu and Xie, 2021]. In particular, we allow unknown dependencies to exist within features and responses that arrive in sequence. Method-wise, ERAPS is a distribution-free and ensemble-based framework that is applicable for arbitrary classifiers. Theoretically, we bound the coverage gap without assuming data exchangeability and show asymptotic set convergence. Empirically, we demonstrate valid marginal and conditional coverage by ERAPS, which also tends to yield smaller prediction sets than competing methods.

Motivation & Objective

• To develop a conformal prediction framework for time-series classification that does not rely on data exchangeability, a restrictive assumption in sequential data.
• To construct prediction sets with valid marginal and conditional coverage under arbitrary dependencies in features and responses.
• To improve efficiency and accuracy by leveraging ensemble methods and adaptive non-conformity scoring.
• To theoretically bound coverage gaps and demonstrate asymptotic convergence without i.i.d. or exchangeability assumptions.
• To empirically validate the method on real-world time-series datasets, showing smaller set sizes and robust coverage.

Proposed method

• ERAPS uses an ensemble of arbitrary classifiers to estimate class probabilities, enabling robustness to model uncertainty.
• It computes non-conformity scores based on predicted class probabilities, using a rolling window of past observations to estimate the empirical distribution of these scores.
• Prediction sets are constructed by including all classes whose non-conformity scores fall below the threshold defined by the $(1 - \alpha)$-quantile of the empirical distribution of past non-conformity scores.
• The method employs a weighting scheme that accounts for temporal dependencies, improving performance in non-exchangeable settings.
• Theoretical analysis bounds the coverage gap using total variation distances and establishes asymptotic convergence of prediction sets to the true conditional coverage level.
• The framework is agnostic to the choice of base classifier, making it applicable to logistic regression, random forests, and deep neural networks.

Experimental results

Research questions

• RQ1 Can conformal prediction be effectively extended to time-series classification without assuming data exchangeability?
• RQ2 Can an ensemble-based approach improve the accuracy and size of prediction sets in time-series classification under temporal dependencies?
• RQ3 Does ERAPS achieve valid marginal and conditional coverage in non-exchangeable, dependent time-series settings?
• RQ4 How does ERAPS compare in set size and coverage validity to existing conformal prediction methods like SAPS and Naive?
• RQ5 What theoretical guarantees can be provided for coverage gap and set convergence under general dependence structures?

Key findings

• ERAPS achieves valid marginal and conditional coverage across all tested datasets, including those with complex temporal dependencies.
• The method consistently produces smaller prediction sets than SAPS and Naive, reducing set size by up to 30% in some strata without sacrificing coverage.
• Conditional coverage is maintained across most strata, with minor under-coverage observed only in rare outlier cases (e.g., α=0.15 on crop data).
• Theoretical analysis shows that the coverage gap is bounded without assuming exchangeability, and prediction sets converge asymptotically to the true coverage level.
• Empirical results confirm that ERAPS outperforms non-ensemble baselines in both coverage validity and set size efficiency.
• The framework is robust to model choice, maintaining strong performance across diverse base classifiers including deep neural networks.