[Paper Review] Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Youngseog Chung, Ian Char+3 more|arXiv (Cornell University)|Sep 21, 2021

Explainable Artificial Intelligence (XAI)27 references40 citations

TL;DR

The paper introduces Uncertainty Toolbox, an open-source Python library for assessing, visualizing, and recalibrating uncertainty quantification (UQ) in regression, using a suite of metrics and visualizations to compare different loss objectives for probabilistic neural networks.

ABSTRACT

With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.

Motivation & Objective

• Motivate robust evaluation of predictive uncertainty in regression settings to enable fair comparisons across UQ methods.
• Provide a unified toolbox with calibrated metrics, recalibration utilities, and visualization tools for distributional predictions.
• Demonstrate that multiple evaluation metrics are needed for a holistic UQ assessment and show how recalibration can improve calibration.
• Offer educational resources (glossary and paper recommendations) to help newcomers and researchers use UQ tools effectively.

Proposed method

• Describe the four main functionalities of Uncertainty Toolbox: evaluation metrics, recalibration via isotonic regression, visualizations, and pedagogy resources.
• Summarize and formalize evaluation metrics for predictive UQ, including calibration, group calibration, sharpness, and proper scoring rules.
• Demonstrate toolbox usage through a case study training probabilistic neural networks with different loss functions and evaluating them with multiple metrics.
• Implement recalibration procedures to improve average calibration of distributional predictions.
• Provide a case-study workflow with a synthetic dataset, multiple loss functions (NLL, CRPS, Check, Interval), and a suite of UQ diagnostics.
• Compare methods using a suite of metrics to illustrate the potential conflicts and complementary information among metrics.

Experimental results

Research questions

• RQ1 How can a unified toolbox improve the comparability of uncertainty quantification in regression tasks?
• RQ2 What are the strengths and limitations of common UQ evaluation metrics across different loss objectives?
• RQ3 Can recalibration (e.g., isotonic regression) improve average calibration of distributional predictions in probabilistic neural networks?
• RQ4 Do multiple metrics provide complementary insights into UQ quality beyond single-score evaluations?

Key findings

• A suite of calibration, sharpness, and proper scoring metrics provides a more nuanced view of UQ quality than any single metric.
• Different loss objectives (NLL, CRPS, Check, Interval) yield distinct calibration and sharpness behaviors, with no single method dominating across all metrics.
• Adversarial group calibration and other calibration diagnostics reveal that some methods are better calibrated for subsets of the input domain, not just on average.
• Recalibration via isotonic regression can improve average calibration of distributional predictions.
• Proper scoring rules capture trade-offs between calibration and sharpness and may rank methods differently depending on the metric used.