[Paper Review] Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder

Zhisheng Xiao, Qing Yan+1 more|arXiv (Cornell University)|Mar 6, 2020

Adversarial Robustness in Machine LearningComputer Science52 references64 citations

TL;DR

The paper introduces Likelihood Regret, an OOD detection score for VAEs, by comparing the best per-sample posterior configuration against the learned population ELBO, and shows LR outperforms likelihood-based OOD detection across several image datasets.

ABSTRACT

Deep probabilistic generative models enable modeling the likelihoods of very high dimensional data. An important application of generative modeling should be the ability to detect out-of-distribution (OOD) samples by setting a threshold on the likelihood. However, some recent studies show that probabilistic generative models can, in some cases, assign higher likelihoods on certain types of OOD samples, making the OOD detection rules based on likelihood threshold problematic. To address this issue, several OOD detection methods have been proposed for deep generative models. In this paper, we make the observation that many of these methods fail when applied to generative models based on Variational Auto-encoders (VAE). As an alternative, we propose Likelihood Regret, an efficient OOD score for VAEs. We benchmark our proposed method over existing approaches, and empirical results suggest that our method obtains the best overall OOD detection performances when applied to VAEs.

Motivation & Objective

• Motivate reliable OOD detection for VAEs where likelihoods can be misleading.
• Propose a per-sample optimization-based score (Likelihood Regret) that mitigates likelihood misalignment.
• Evaluate LR against existing OOD scores on diverse image datasets.
• Analyze robustness of LR across VAE variants and capacities.

Proposed method

• Define Likelihood Regret (LR) as LR(x)=L(x;θ*,τ̂(x))−L(x;θ*,φ*), where L is ELBO-based log-likelihood.
• Compute L by estimating the IWELBO (K samples) for the VAE, then optimize the variational parameters τ to maximize L for a single input with θ* fixed.
• Use either optimizing the encoder φ or directly optimizing τ(x) to obtain τ̂(x).
• Regularize optimization via the VAE bottleneck by restricting changes to the latent posterior parameters.
• Compare LR to baselines (Likelihood, IC, Likelihood Ratio, LMD) across multiple OOD tasks.

Experimental results

Research questions

• RQ1 Can LR reliably distinguish in-distribution from OOD samples for VAEs where standard likelihood fails?
• RQ2 How does LR compare to existing OOD scores on various in-distribution/out-of-distribution pairs?
• RQ3 Is LR robust to different VAE capacities and to β-VAE settings?
• RQ4 What are the computational trade-offs of LR relative to other OOD methods?

Key findings

• LR corrects likelihood misalignment observed with VAEs, delivering high AUC-ROC on most OOD tasks.
• On Fashion MNIST vs MNIST, LR improves AUC-ROC from 0.165 (likelihood) to 0.999.
• On CIFAR-10 vs SVHN, LR improves AUC-ROC from 0.161 (likelihood) to 0.876.
• LR variants optimized over encoder (LR_E) and over latent statistics (LR_Z) both perform well, with LR_E generally better.
• LR is robust across β-VAE settings and VAEs of varying capacity, though very large capacity can slightly reduce performance on some tasks.