[Paper Review] Deep Deterministic Uncertainty: A Simple Baseline

Jishnu Mukhoti, Andreas Kirsch+3 more|arXiv (Cornell University)|Feb 23, 2021

Fault Detection and Control Systems18 citations

TL;DR

This paper introduces Deep Deterministic Uncertainty (DDU), a simple yet powerful baseline for uncertainty quantification in deep learning. By leveraging a well-regularized feature space from residual connections and spectral normalization, DDU achieves state-of-the-art performance in out-of-distribution detection and active learning using a single forward pass, outperforming complex methods like DUQ and SNGP without fine-tuning or ensembling.

ABSTRACT

Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive. We take two complex single-forward-pass uncertainty approaches, DUQ and SNGP, and examine whether they mainly rely on a well-regularized feature space. Crucially, without using their more complex methods for estimating uncertainty, a single softmax neural net with such a feature-space, achieved via residual connections and spectral normalization, *outperforms* DUQ and SNGP's epistemic uncertainty predictions using simple Gaussian Discriminant Analysis *post-training* as a separate feature-space density estimator -- without fine-tuning on OoD data, feature ensembling, or input pre-procressing. This conceptually simple *Deep Deterministic Uncertainty (DDU)* baseline can also be used to disentangle aleatoric and epistemic uncertainty and performs as well as Deep Ensembles, the state-of-the art for uncertainty prediction, on several OoD benchmarks (CIFAR-10/100 vs SVHN/Tiny-ImageNet, ImageNet vs ImageNet-O) as well as in active learning settings across different model architectures, yet is *computationally cheaper*.

Motivation & Objective

• To investigate whether complex single-forward-pass uncertainty methods like DUQ and SNGP rely primarily on a well-regularized feature space.
• To develop a conceptually simple baseline that achieves strong uncertainty estimation without ensembling, fine-tuning, or input preprocessing.
• To disentangle aleatoric and epistemic uncertainty using a minimal architecture with well-regularized features.
• To evaluate the method on out-of-distribution detection and active learning benchmarks across diverse architectures.

Proposed method

• Train a standard softmax neural network with residual connections and spectral normalization to produce a well-regularized feature space.
• Use Gaussian Mixture Models (GMM) as a post-training density estimator on the final-layer features to estimate epistemic uncertainty.
• Use softmax cross-entropy on the logits to estimate aleatoric uncertainty.
• Apply Gaussian Discriminant Analysis (GDA) to jointly model class-conditional densities for disentangling uncertainty types.
• Train the model end-to-end with standard cross-entropy loss, then apply GMM and softmax entropy post-inference.
• Avoid fine-tuning on OoD data or input pre-processing, relying solely on architectural regularization and post-hoc density estimation.

Experimental results

Research questions

• RQ1 Can a simple, deterministic single-forward-pass model outperform complex uncertainty methods like DUQ and SNGP when using only a well-regularized feature space?
• RQ2 Does the performance of advanced uncertainty methods stem primarily from feature space regularization rather than complex uncertainty estimation mechanisms?
• RQ3 Can a minimal architecture with residual connections and spectral normalization enable effective disentanglement of aleatoric and epistemic uncertainty?
• RQ4 How does the proposed DDU baseline compare to Deep Ensembles in out-of-distribution detection and active learning across multiple datasets and architectures?
• RQ5 What is the impact of architectural inductive biases (e.g., residual connections, spectral normalization) on the quality of feature-space density estimation for uncertainty quantification?

Key findings

• DDU achieves state-of-the-art out-of-distribution detection performance on CIFAR-10 vs SVHN/CIFAR-100/Tiny-ImageNet and ImageNet vs ImageNet-O benchmarks, with AUROC scores of 97.86% and 91.34% respectively.
• On CIFAR-100 vs SVHN/Tiny-ImageNet, DDU achieves 87.53% AUROC for SVHN and 83.13% for Tiny-ImageNet, outperforming SNGP and matching the performance of 5-ensemble models.
• DDU reduces expected calibration error (ECE) to 0.85±0.04 on CIFAR-10 vs SVHN, outperforming DUQ (1.55±0.08) and matching the best-performing method.
• In active learning, DDU performs as well as Deep Ensembles across multiple architectures, demonstrating strong uncertainty calibration and predictive power.
• The method successfully disentangles aleatoric and epistemic uncertainty: softmax entropy captures ambiguity in in-distribution data, while GMM on features captures model uncertainty for OoD samples.
• Without proper inductive biases (e.g., residual connections and spectral normalization), feature-space density estimation fails due to feature collapse, causing overlap between in-distribution and out-of-distribution densities.