[Paper Review] Input complexity and out-of-distribution detection with likelihood-based generative models

Joan Serrà, David Álvarez+4 more|arXiv (Cornell University)|Sep 25, 2019

Adversarial Robustness in Machine LearningComputer Science30 references98 citations

TL;DR

The paper shows that input complexity biases likelihoods from generative models, and proposes a likelihood-ratio like OOD score S = -log p(x|M) - L(x) using a complexity estimate L(x) from lossless compression, yielding competitive OOD detection across diverse datasets and models without hyperparameters.

ABSTRACT

Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

Motivation & Objective

• Demonstrate that input complexity strongly biases likelihoods from generative models.
• Propose a complexity-adjusted OOD score to detect out-of-distribution inputs.
• Compare the proposed score to existing OOD methods across diverse datasets and models.
• Provide a theoretical interpretation of the score as a likelihood-ratio test.
• Show practical, parameter-free applicability of the method.

Proposed method

• Train autoregressive and invertible generative models (PixelCNN++ and Glow) on CIFAR10/FashionMNIST-like datasets to obtain log-likelihoods.
• Estimate input complexity L(x) via normalized compressed length using lossless compressors (PNG, JPEG2000, FLIF).
• Define OOD score S(x) = -ell_M(x) - L(x), where ell_M(x) is the log-likelihood under the model M.
• Interpret S(x) as a likelihood-ratio test statistic under a universal compressor M0.
• Evaluate S on multiple in-distribution vs. out-of-distribution pairs using AUROC as the primary metric.
• Explore impact of model size and compressor choice on S’s performance.

Experimental results

Research questions

• RQ1 Can input complexity explain the failure of likelihood-based OOD detection with generative models?
• RQ2 Does a complexity-adjusted likelihood score improve OOD detection across diverse datasets and models?
• RQ3 How does the proposed score compare to existing generative- and classifier-based OOD methods?
• RQ4 Is the score parameter-free and robust to model size and compressor choice?
• RQ5 Can the approach be interpreted within Bayesian model comparison or MDL principles?

Key findings

• Likelihoods from generative models are strongly negatively correlated with input complexity, sometimes explaining most of the variance in log-likelihoods.
• A complexity-based adjustment via L(x) yields a parameter-free OOD score S that improves AUROC over using -ell_M(x) alone across many datasets.
• For CIFAR10-trained models, S achieves AUROCs typically above 0.7 and often near 0.9–1.0 on several datasets, outperforming likelihoods alone.
• S is competitive with, and in many cases superior to, existing classifier- and generative-based OOD methods, while using no hyperparameters besides the choice of model and compressor.
• Larger generative models and better compressors tend to improve discrimination performance of S.