[Paper Review] Uncertainty in the Variational Information Bottleneck

Alexander A. Alemi, Ian Fischer+1 more|arXiv (Cornell University)|Jul 2, 2018

Anomaly Detection Techniques and ApplicationsComputer Science8 references65 citations

TL;DR

The paper demonstrates that Variational Information Bottleneck (VIB) yields calibrated predictions and effective out-of-distribution detection on FashionMNIST without extra calibration, by modeling uncertainty in representations and rates.

ABSTRACT

We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.

Motivation & Objective

• Motivate reliable uncertainty quantification in neural networks amid aleatoric, epistemic, and out-of-distribution uncertainty.
• Show that VIB provides calibrated predictions without extra calibration techniques.
• Demonstrate VIB's effectiveness for out-of-distribution detection and robustness on a standard dataset.

Proposed method

• Use a variational bound of the Information Bottleneck to learn a stochastic encoder eθ(z|x) and a variational marginal mφ(z).
• Train a variational classifier qψ(y|z) to predict labels from latent codes Z.
• Optimize a bound that maximizes E[log qψ(y|z)] − β E[log (eθ(z|x)/mφ(z))], blending accuracy and rate.
• Model uncertainty through doubly stochasticity: randomness in both Z and Y, enabling mean and variance in predictions.
• Employ Monte Carlo sampling to estimate expectations during training and testing.

Experimental results

Research questions

• RQ1 Does VIB improve calibration of neural networks compared to deterministic baselines?
• RQ2 Can VIB improve detection of out-of-distribution data without post-hoc calibration?
• RQ3 How do VIB’s uncertainty metrics (e.g., rate R and entropy H) relate to error and OOD detection performance?

Key findings

• VIB yields better calibration than a baseline deterministic classifier, reducing overconfidence on test data.
• On FashionMNIST, VIB achieves accuracy around 92.8–92.9% with competitive error-detection metrics compared to the baseline and temperature-scaled baselines.
• VIB provides two uncertainty signals—rate (R) and entropy (H)—that aid out-of-distribution and error detection, with R performing strongly for OOD detection.
• Temperature scaling remains a strong post-hoc method, but VIB can improve calibration and OOD detection without temperature scaling, and R can outperform H for certain OoD signals.
• VIB's uncertainty signals can detect both gross and subtle distributional shifts, including mirrored or rotated versions of FashionMNIST data.