[Paper Review] Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality

Eric Nalisnick, Akihiro Matsukawa+2 more|arXiv (Cornell University)|Jun 7, 2019

Generative Adversarial Networks and Image Synthesis65 references58 citations

TL;DR

The paper introduces a model-agnostic, bootstrap-based typicality test to detect out-of-distribution inputs for deep generative models by testing membership in the model’s typical set using input likelihoods.

ABSTRACT

Recent work has shown that deep generative models can assign higher likelihood to out-of-distribution data sets than to their training data (Nalisnick et al., 2019; Choi et al., 2019). We posit that this phenomenon is caused by a mismatch between the model's typical set and its areas of high probability density. In-distribution inputs should reside in the former but not necessarily in the latter, as previous work has presumed. To determine whether or not inputs reside in the typical set, we propose a statistically principled, easy-to-implement test using the empirical distribution of model likelihoods. The test is model agnostic and widely applicable, only requiring that the likelihood can be computed or closely approximated. We report experiments showing that our procedure can successfully detect the out-of-distribution sets in several of the challenging cases reported by Nalisnick et al. (2019).

Motivation & Objective

• Motivate the misalignment between a model's high-density regions and its typical set as a source of misleading likelihoods for OOD data.
• Define a principled, scalable GoF test that decides OOD without relying on CDFs or low-dimensional projections.
• Provide a practical implementation that works with any DGM with computable likelihood and is robust in low-data regimes.
• Evaluate the method across diverse DGMs and data sets to identify both successes and failure modes of typicality-based OOD detection.

Proposed method

• Formalize typicality via Shannon entropy and the (ε,N)-typical set definition.
• Rewrite typicality as a GoF test: decide OOD by checking if a test batch lies in AεM[p(x;θ)].
• Estimate model entropy with either a resubstitution estimator or Monte Carlo samples: H[p(x;θ)] ≈ (1/S)∑−log p(x̂s;θ) or (1/N)∑−log p(xn;θ).
• Set the decision threshold ε using bootstrap confidence intervals on held-out validation data to control type-I error.
• Compute the test statistic ε̂ = | (1/M)∑−log p(x̃m;θ) − H[p(x;θ)] | and reject if ε̂ > ε.
• Provide a fully offline bootstrap procedure to precompute the threshold εαM for a desired α before test-time.

Experimental results

Research questions

• RQ1 Can a typicality-based criterion reliably distinguish in-distribution from out-of-distribution data for deep generative models?
• RQ2 Does testing membership in the model’s typical set outperform density-based OOD detection in high-dimensional settings?
• RQ3 How practical and robust is a bootstrap-derived threshold for GoF testing in real DGM applications?
• RQ4 What are the failure modes and limits of typicality-based OOD detection across different architectures and datasets?

Key findings

• The proposed typicality test can detect OOD inputs in several challenging cases where likelihood alone misleads, as shown across multiple DGMs and data sets.
• Using bootstrap to set the OOD threshold yields a robust decision rule that works with high-dimensional data and various model classes (normalizing flows, VAEs, autoregressive models).
• Empirical entropy estimation via resubstitution often improves OOD detection performance compared to purely Monte Carlo estimates.
• The method reveals notable failure modes and variability across different model-data combinations, highlighting scope for future improvements.
• Compared against several GoF baselines (t-test, KS-test, MMD, KSD, annulus), the typicality approach demonstrates competitive and sometimes superior performance in distinguishing OOD batches.