[Paper Review] Single-Model Uncertainties for Deep Learning

Nataša Tagasovska, David López-Paz|arXiv (Cornell University)|Nov 2, 2018

Adversarial Robustness in Machine LearningComputer Science80 citations

TL;DR

The paper introduces two single-model uncertainty estimators for deep learning: Simultaneous Quantile Regression (SQR) for aleatoric uncertainty and Orthonormal Certificates (OCs) for epistemic uncertainty, achieving competitive results without ensembling or retraining.

ABSTRACT

We provide single-model estimates of aleatoric and epistemic uncertainty for deep neural networks. To estimate aleatoric uncertainty, we propose Simultaneous Quantile Regression (SQR), a loss function to learn all the conditional quantiles of a given target variable. These quantiles can be used to compute well-calibrated prediction intervals. To estimate epistemic uncertainty, we propose Orthonormal Certificates (OCs), a collection of diverse non-constant functions that map all training samples to zero. These certificates map out-of-distribution examples to non-zero values, signaling epistemic uncertainty. Our uncertainty estimators are computationally attractive, as they do not require ensembling or retraining deep models, and achieve competitive performance.

Motivation & Objective

• Provide single-model estimates of aleatoric and epistemic uncertainty in deep neural networks.
• Develop a scalable method to obtain prediction intervals without model ensembling or retraining.
• Offer a unified discussion of related uncertainty estimation approaches to position the proposed methods.

Proposed method

• Simultaneous Quantile Regression (SQR) minimizes a randomized pinball loss to learn all conditional quantiles jointly, enabling non-Gaussian, skewed, and heteroskedastic aleatoric uncertainty.
• SQR yields prediction intervals from the estimated quantiles via u_a(x*) = f(x*,1−α/2) − f(x*,α/2).
• Orthonormal Certificates (OCs) construct a set of diverse, non-constant certificate functions C that map training representations φ(x) to zero, enabling detection of out-of-domain samples via u_e(x*) = ||C^T φ(x*)||^2.
• Certificates are trained with an orthonormality constraint to ensure diversity, implemented as minimizing a loss with a regularizer ||C^T C − I_k||.
• OC performance is analyzed through tail bounds showing separation between in-domain and out-domain distributions.
• The approach is compatible with any pre-trained network by stacking on the last-layer representation.

Experimental results

Research questions

• RQ1 Can a single neural network model provide reliable, calibrated prediction intervals for aleatoric uncertainty without ensembling?
• RQ2 Can a single-model approach detect epistemic uncertainty and out-of-distribution samples effectively without retraining or extensive hyperparameter tuning?
• RQ3 How do SQR and OC methods compare to existing uncertainty estimation approaches in practice on standard benchmarks?
• RQ4 What theoretical guarantees or insights support the behavior of orthonormal certificates in distinguishing in-domain and out-of-domain data?

Key findings

• SQR delivers well-calibrated 1−α prediction intervals and can model complex, non-Gaussian aleatoric uncertainty without model ensembling.
• OCs provide competitive out-of-distribution detection performance using a single model, often outperforming several baselines across multiple datasets.
• SQR can be added as an extra output layer to any neural network without compromising performance or requiring retraining.
• OC theoretical analysis includes tail bounds that describe the behavior of epistemic uncertainty for in-domain versus out-of-domain data.
• The proposed methods achieve competitive results while reducing computation and storage relative to ensembles.