[Paper Review] Uncertainty Sets for Image Classifiers using Conformal Prediction

Anastasios N. Angelopoulos, Stephen Bates+2 more|arXiv (Cornell University)|Sep 29, 2020

Anomaly Detection Techniques and ApplicationsComputer Science35 references31 citations

TL;DR

The paper introduces Regularized Adaptive Prediction Sets (RAPS), a conformal-calibration-based method that turns any image classifier into predictive sets with finite-sample coverage guarantees, producing smaller and more adaptive uncertainty sets than naive or APS approaches, demonstrated on ImageNet with various CNN backbones.

ABSTRACT

Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving coverage with sets that are often factors of 5 to 10 smaller than a stand-alone Platt scaling baseline.

Motivation & Objective

• Motivate the need for actionable uncertainty quantification in image classification beyond point estimates.
• Develop a fast, model-agnostic method to produce predictive uncertainty sets with finite-sample coverage guarantees.
• Improve upon existing conformal prediction approaches by stabilizing tail probabilities to yield smaller, more adaptive sets.
• Demonstrate practical performance gains on large-scale datasets (ImageNet, ImageNet-V2) across multiple CNN architectures.

Proposed method

• Start from a pre-trained image classifier producing class probabilities.
• Order classes by predicted probability and introduce a regularization term to penalize inclusion of unlikely tail classes.
• Define a conformal calibration scheme that selects a tuning parameter τ to guarantee 1−α coverage on a calibration set (split-conformal prediction).
• Use a regularized predictive set C*(x,u,τ) that combines rank-based mass, randomized thresholding, and a regularization penalty controlled by k_reg and λ.
• Calibrate τ with a holdout conformal set to ensure finite-sample coverage for future test points (Theorem 1 and Proposition 1).
• Prove that RAPS dominates fixed-size top-k sets and provides coverage within a 1/(n+1) slack (Proposition 2).
• Evaluate on ImageNet-Val and ImageNet-V2 across multiple architectures; compare against naive and APS baselines.

Experimental results

Research questions

• RQ1 Can conformal prediction be deployed on modern CNN classifiers to provide guaranteed predictive sets with finite-sample coverage?
• RQ2 Does incorporating regularization into the conformal prediction scheme yield smaller, more adaptive uncertainty sets without sacrificing coverage?
• RQ3 How does RAPS compare to naive and Adaptive Prediction Sets (APS) in terms of coverage, set size, and adaptiveness across different architectures and datasets?
• RQ4 What is the impact of tuning parameters (k_reg, λ) on the adaptiveness and coverage of the resulting predictive sets?
• RQ5 Do the proposed methods maintain guarantees under distribution shifts (e.g., Imagenet vs. Imagenet-V2) when using a calibration set drawn from the target distribution?

Key findings

• RAPS achieves guaranteed 1−α coverage while producing substantially smaller predictive sets than naïve or APS approaches.
• Across nine CNN models on ImageNet-Val, RAPS substantially reduces average set size while maintaining near-target coverage, outperforming APS in both size and stability.
• On Imagenet-V2, RAPS maintains coverage with similarly small average set sizes, despite distribution shift between training and evaluation data.
• RAPS sets are smaller for easy images and larger for hard ones, demonstrating adaptive behavior that tailors uncertainty to instance difficulty.
• Automated tuning of k_reg and λ can improve adaptiveness and maintain coverage, with RAPS outperforming APS in size-stratified coverage balance.
• Theoretical guarantees show RAPS dominates fixed-size top-k predictive sets and that coverage holds with a finite-sample correction term (1/(n+1)).