[Paper Review] WAIC, but Why? Generative Ensembles for Robust Anomaly Detection

Hyunsun Choi, Eric Jang+1 more|arXiv (Cornell University)|Oct 2, 2018

Anomaly Detection Techniques and ApplicationsComputer Science36 references158 citations

TL;DR

The paper introduces Generative Ensembles that use WAIC (Watanabe-Akaike Information Criterion) to robustly detect out-of-distribution inputs by combining density estimates with epistemic uncertainty from ensembles of generative models. It also discusses why likelihood alone can be misleading and provides empirical results across image and credit-card datasets.

ABSTRACT

Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.

Motivation & Objective

• Motivate safe ML deployment by detecting inputs that differ from training data (OoD) without relying on task-specific anomalies.
• Investigate limitations of likelihood-based OoD detection and explore WAIC as a robust alternative.
• Propose Generative Ensembles that estimate epistemic uncertainty to improve density-based anomaly detection.
• Evaluate on image datasets (MNIST, Fashion-MNIST, CIFAR-10) and a real-world credit card fraud dataset to compare against baselines.

Proposed method

• Define WAIC as E_theta[log p_theta(x)] - Var_theta[log p_theta(x)], using an ensemble of independently trained generative models as approximate posterior samples.
• Apply WAIC to likelihood models (autoregressive, flow-based, VAE) to obtain a robust anomaly score.
• Demonstrate how ensemble variance regularizes sensitivity to posterior choice and helps identify OoD inputs.
• Extend GAN-based anomaly detection by leveraging predictive uncertainty from an ensemble of discriminators to detect OoD inputs.
• Experimentally compare Generative Ensembles against ODIN, VIB, and density-based baselines on multiple datasets.

Experimental results

Research questions

• RQ1 Can WAIC-summarized uncertainty from an ensemble of generative models improve OoD detection beyond single-model likelihoods?
• RQ2 Do likelihood-based OoD signals misidentify typical regions in high-dimensional data, and can WAIC mitigate this misbehavior?
• RQ3 How do Generative Ensembles perform relative to discriminative OoD detectors (e.g., ODIN, VIB) on standard benchmarks and real-world anomaly settings?
• RQ4 What practical insights exist for training generative models (e.g., beta-VAE adjustments) to improve OoD detection?

Key findings

• Ensemble-based WAIC can outperform single-model likelihoods on many OoD tasks across MNIST, Fashion-MNIST, CIFAR-10, and other datasets.
• WAIC often identifies OoD samples that have high likelihood under individual models, addressing a known limitation of likelihood-based detection.
• Generative Ensembles are competitive with discriminative OoD baselines (ODIN, VIB) and can outperform them on several tasks without label information.
• Adjustments to training objectives (e.g., beta-VAE) can affect the quality of OoD signals, with certain settings improving detection.
• On a real-world Kaggle Credit Fraud dataset, density-based WAIC-based methods show strong performance compared to a discriminative classifier baseline when trained only on normal transactions.