[Paper Review] Probabilistic Conformal Prediction Using Conditional Random Samples

Zhendong Wang, Ruijiang Gao+3 more|arXiv (Cornell University)|Jun 14, 2022

Neural Networks and Applications4 citations

TL;DR

This paper introduces Probabilistic Conformal Prediction (PCP), a method that constructs sharp, valid predictive sets for multimodal and multi-target regression by sampling from a conditional generative model and forming union-of-balls around samples. PCP ensures finite-sample marginal coverage and outperforms existing conformal methods in sharpness, especially on complex, non-continuous target distributions like taxi drop-off locations.

ABSTRACT

This paper proposes probabilistic conformal prediction (PCP), a predictive inference algorithm that estimates a target variable by a discontinuous predictive set. Given inputs, PCP construct the predictive set based on random samples from an estimated generative model. It is efficient and compatible with either explicit or implicit conditional generative models. Theoretically, we show that PCP guarantees correct marginal coverage with finite samples. Empirically, we study PCP on a variety of simulated and real datasets. Compared to existing methods for conformal inference, PCP provides sharper predictive sets.

Motivation & Objective

• To address the limitation of existing conformal prediction methods that produce continuous, overly broad predictive sets when the target distribution is multimodal.
• To develop a method that maintains finite-sample marginal coverage while producing sharper, more informative predictive sets by adapting to the target's density landscape.
• To enable compatibility with both explicit and implicit conditional generative models, including likelihood-free settings.
• To extend the framework to multi-target regression by capturing dependencies between targets through high-density set construction.
• To empirically demonstrate superior sharpness and valid coverage across synthetic and real-world datasets, including NYC taxi drop-off prediction.

Proposed method

• PCP splits the data into preliminary and calibration sets to train a conditional generative model on the preliminary set.
• For each calibration point, it generates K independent samples from the fitted model and computes nonconformity scores as the distance between each sample and the true label.
• It then computes the (1−α) empirical quantile of these nonconformity scores to define a radius for predictive balls.
• For a new input, PCP generates K samples from the model, forms balls of radius equal to the calibration quantile around each, and takes their union as the predictive set.
• High-density PCP (HD-PCP) further refines the set by selecting only the most probable samples, improving sparsity and interpretability.
• The method is compatible with implicit models, requiring only the ability to generate random samples from the conditional distribution.

Experimental results

Research questions

• RQ1 Can a conformal prediction method produce valid and sharp predictive sets for multimodal target distributions where continuous intervals are suboptimal?
• RQ2 How can conformal prediction be adapted to work efficiently with implicit or likelihood-free conditional generative models?
• RQ3 Does the union-of-balls approach based on random samples from a generative model maintain finite-sample marginal coverage?
• RQ4 How does the performance of PCP compare to existing conformal methods in terms of set size and coverage on real-world and synthetic data?
• RQ5 Can PCP and HD-PCP effectively model dependencies in multi-target regression while maintaining sharpness and validity?

Key findings

• PCP achieves valid marginal coverage with finite samples, as theoretically proven, ensuring the true label is included in the predictive set with probability at least 1−α.
• On the NYC taxi dataset, PCP and HD-PCP produce significantly sharper predictive sets than CDSplit and CHR, capturing distinct neighborhoods and airports without including low-density regions between them.
• In multi-target regression, HD-PCP reduces predictive set size by up to 50% compared to CDSplit and CHR, with 83% of pairwise comparisons showing HD-PCP as sharper.
• For synthetic data with increasing correlation ρ, PCP and HD-PCP maintain or reduce set size, while other baselines remain largely unchanged, indicating adaptability to target structure.
• PCP achieves a mean set size of 171.61 (HD-PCP) and 205.63 (PCP) at ρ=9, compared to 415.86 (CDSplit-MixD) and 689.22 (CHR-NNet), demonstrating superior sharpness.
• Empirical results show PCP and HD-PCP are robust across diverse data types, including multimodal, multi-target, and high-dimensional regression tasks.