[Paper Review] Calibration of Model Uncertainty for Dropout Variational Inference

Max-Heinrich Laves, Sontje Ihler+2 more|arXiv (Cornell University)|Jun 20, 2020

Adversarial Robustness in Machine Learning34 references11 citations

TL;DR

This paper proposes logit scaling techniques—temperature, vector, and auxiliary scaling—to recalibrate model uncertainty from Monte Carlo dropout variational inference, significantly reducing miscalibration as measured by Expected Uncertainty Calibration Error (UCE). The method improves robust out-of-distribution detection and reliable rejection of uncertain predictions across CIFAR-10/100 and SVHN with state-of-the-art CNNs.

ABSTRACT

The model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. In this paper, different logit scaling methods are extended to dropout variational inference to recalibrate model uncertainty. Expected uncertainty calibration error (UCE) is presented as a metric to measure miscalibration. The effectiveness of recalibration is evaluated on CIFAR-10/100 and SVHN for recent CNN architectures. Experimental results show that logit scaling considerably reduce miscalibration by means of UCE. Well-calibrated uncertainty enables reliable rejection of uncertain predictions and robust detection of out-of-distribution data.

Motivation & Objective

• To address miscalibration in model uncertainty from Monte Carlo dropout in Bayesian neural networks.
• To introduce a new metric, Expected Uncertainty Calibration Error (UCE), to quantify uncertainty miscalibration.
• To extend frequentist confidence calibration techniques (logit scaling) to Bayesian model uncertainty.
• To empirically validate that calibrated uncertainty enhances out-of-distribution detection and prediction reliability.
• To demonstrate that simple scaling methods outperform complex or training-time regularization techniques.

Proposed method

• Proposes Expected Uncertainty Calibration Error (UCE) as a classwise metric to quantify miscalibration of model uncertainty.
• Adapts temperature, vector, and auxiliary scaling from confidence calibration to recalibrate uncertainty in Monte Carlo dropout variational inference.
• Applies logit scaling to the output logits of Bayesian neural networks trained with Gaussian dropout to adjust epistemic uncertainty estimates.
• Uses a separate calibration set to learn optimal scaling parameters that minimize UCE.
• Employs entropy maximization principles to justify the use of logit scaling for uncertainty calibration.
• Validates calibration using reliability diagrams and evaluates performance on CIFAR-10/100 and SVHN with ResNet, DenseNet, and other architectures.

Experimental results

Research questions

• RQ1 Can logit scaling techniques used for confidence calibration be extended to calibrate model uncertainty in Bayesian neural networks?
• RQ2 How effective is Expected Uncertainty Calibration Error (UCE) in measuring miscalibration of epistemic uncertainty from Monte Carlo dropout?
• RQ3 Does recalibrating uncertainty via logit scaling improve out-of-distribution detection compared to uncalibrated MC dropout?
• RQ4 How does uncertainty calibration affect the reliability of rejecting uncertain predictions?
• RQ5 Do simple scaling methods (e.g., temperature scaling) outperform more complex or training-time regularization techniques like confidence penalty?

Key findings

• Logit scaling significantly reduces Expected Uncertainty Calibration Error (UCE), with temperature scaling performing nearly as well as more complex methods.
• Calibrated uncertainty enables robust rejection of uncertain predictions, achieving top-1 error below 1% when using temperature or vector scaling.
• On ResNet and DenseNet, calibrated uncertainty maintains high uncertainty levels for out-of-distribution data (>50%), enabling reliable OoD detection.
• Confidence penalty leads to overestimation of uncertainty and often worsens calibration, especially on larger models.
• The Bayesian likelihood from MC dropout is inherently better calibrated than frequentist confidence, and logit scaling further improves this baseline.
• Vector and auxiliary scaling improve uncertainty calibration but can negatively affect predictive accuracy, unlike temperature scaling which maintains accuracy.