[Paper Review] Uncertainty Estimation Using a Single Deep Deterministic Neural Network

Joost van Amersfoort, Lewis Smith+2 more|arXiv (Cornell University)|Mar 4, 2020

Adversarial Robustness in Machine Learning37 references233 citations

TL;DR

DUQ trains a deterministic deep model with class centroids and an RBF-like kernel to estimate predictive uncertainty in a single forward pass, enabling OoD detection and competitive accuracy with a gradient-regularized Jacobian.

ABSTRACT

We propose a method for training a deterministic deep model that can find and reject out of distribution data points at test time with a single forward pass. Our approach, deterministic uncertainty quantification (DUQ), builds upon ideas of RBF networks. We scale training in these with a novel loss function and centroid updating scheme and match the accuracy of softmax models. By enforcing detectability of changes in the input using a gradient penalty, we are able to reliably detect out of distribution data. Our uncertainty quantification scales well to large datasets, and using a single model, we improve upon or match Deep Ensembles in out of distribution detection on notable difficult dataset pairs such as FashionMNIST vs. MNIST, and CIFAR-10 vs. SVHN.

Motivation & Objective

• Aim to train a deterministic model that can detect out-of-distribution data in a single forward pass.
• Stabilize training of centroid-based RBF-like networks to match softmax accuracy.
• Enable reliable OoD detection by enforcing sensitivity to input changes via Jacobian regularization.
• Investigate how gradient penalties affect uncertainty quality and generalization.
• Evaluate DUQ on standard OoD benchmarks and assess hyperparameter effects.

Proposed method

• Represent inputs with a deep feature extractor f_theta, followed by class-specific centroids e_c and a per-class linear mapping W_c that define an RBF-like kernel to each centroid.
• Prediction is made by selecting the closest centroid via K_c = exp(-||W_c f_theta(x) - e_c||^2 / (2 sigma^2)) and uncertainty is the distance to the closest centroid.
• Train with a multivariate Bernoulli-style loss that minimizes the distance to the correct centroid while maximizing distances to others (one-vs-rest).
• Update class centroids e_c online via exponential moving averages of the class-specific features with momentum gamma to stabilize training.
• Regularize the Jacobian (gradient of outputs with respect to input) with a gradient penalty to enforce a Lipschitz constraint and prevent feature collapse.
• Introduce two-sided gradient penalty that targets the norm of the gradient of sum_c K_c, improving sensitivity and OoD detection over one-sided penalties.
• Compare to softmax baselines and Deep Ensembles, showing competitive accuracy with improved single-pass uncertainty estimation.

Experimental results

Research questions

• RQ1 Can a deterministic, centroid-based deep model provide reliable predictive uncertainty in a single forward pass without relying on ensembles or MC sampling?
• RQ2 How does gradient-based Jacobian regularization affect uncertainty quality and feature collapse in RBF-like network architectures?
• RQ3 What is the impact of hyperparameters (length scale sigma and gradient penalty lambda) on accuracy and OoD detection performance across datasets?
• RQ4 How does DUQ perform on challenging OoD benchmarks such as FashionMNIST vs MNIST and CIFAR-10 vs SVHN compared to Deep Ensembles and other baselines?
• RQ5 Is DUQ able to distinguish epistemic vs. aleatoric uncertainty, and how are centroids positioned in feature space to reflect these uncertainties?

Key findings

• DUQ achieves competitive accuracy with softmax baselines on standard datasets (e.g., FashionMNIST and CIFAR-10) while providing single-pass uncertainty estimates.
• Two-sided gradient penalty on the Jacobian improves out-of-distribution detection and prevents feature collapse, outperforming one-sided penalties in OoD tasks.
• On FashionMNIST vs MNIST, DUQ with gradient penalty achieves higher AUROC for separating in-distribution from out-of-distribution data than several baselines, with lambda=0.05–0.1 giving strong performance.
• DUQ can separate OoD data (e.g., MNIST, NotMNIST) from in-distribution data with AUROC values around 0.95 in FashionMNIST experiments and maintains competitive accuracy (e.g., ~92.4% on FashionMNIST).
• On CIFAR-10 vs SVHN, DUQ shows competitive OoD detection performance with a single model, though Deep Ensembles can outperform in some complex datasets; the gradient penalty substantially boosts OoD detection compared to no penalty.
• Training and inference costs are significantly lower for DUQ than Deep Ensembles, especially at test time, since DUQ requires only a single forward pass for uncertainty estimation.