[Paper Review] Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Features

Robin Tibor Schirrmeister, Yuxuan Zhou+2 more|arXiv (Cornell University)|Jun 18, 2020

Anomaly Detection Techniques and ApplicationsComputer Science36 references31 citations

TL;DR

The paper analyzes why likelihood-based anomaly detection with invertible generative models is biased by low-level features and proposes two hierarchy-based detection methods: likelihood ratios across distributions and using final-scale likelihood contributions, with strong unsupervised performance and competitive supervised results.

ABSTRACT

Deep generative networks trained via maximum likelihood on a natural image dataset like CIFAR10 often assign high likelihoods to images from datasets with different objects (e.g., SVHN). We refine previous investigations of this failure at anomaly detection for invertible generative networks and provide a clear explanation of it as a combination of model bias and domain prior: Convolutional networks learn similar low-level feature distributions when trained on any natural image dataset and these low-level features dominate the likelihood. Hence, when the discriminative features between inliers and outliers are on a high-level, e.g., object shapes, anomaly detection becomes particularly challenging. To remove the negative impact of model bias and domain prior on detecting high-level differences, we propose two methods, first, using the log likelihood ratios of two identical models, one trained on the in-distribution data (e.g., CIFAR10) and the other one on a more general distribution of images (e.g., 80 Million Tiny Images). We also derive a novel outlier loss for the in-distribution network on samples from the more general distribution to further improve the performance. Secondly, using a multi-scale model like Glow, we show that low-level features are mainly captured at early scales. Therefore, using only the likelihood contribution of the final scale performs remarkably well for detecting high-level feature differences of the out-of-distribution and the in-distribution. This method is especially useful if one does not have access to a suitable general distribution. Overall, our methods achieve strong anomaly detection performance in the unsupervised setting, and only slightly underperform state-of-the-art classifier-based methods in the supervised setting. Code can be found at https://github.com/boschresearch/hierarchical_anomaly_detection.

Motivation & Objective

• Explain why deep generative networks misidentify anomalies due to domain priors and low-level feature dominance.
• Propose methods to mitigate model bias using a hierarchy of distributions and a hierarchy of features.
• Demonstrate anomaly detection performance in unsupervised and supervised settings on image datasets.
• Provide practical guidance and code for hierarchical anomaly detection.
• Extend the analysis to non-image domains (medical MRI) and discuss broader impact.

Proposed method

• Baseline use of Glow/invertible networks trained by maximum likelihood on natural images.
• Propose hierarchy-of-distributions: compare in-distribution model p_in with general-distribution model p_g via log-likelihood ratio as anomaly score.
• Derive an outlier loss L_o that uses log-likelihood ratios on general-distribution samples to improve training.
• Leverage hierarchy-of-features: show that final-scale likelihood contributions c_3(x) dominate and can serve as an anomaly score.
• Extend to supervised setting with class-conditioned in-distribution model and negative samples from other classes.
• Demonstrate that using final-scale likelihoods can be effective when a general distribution is unavailable.

Experimental results

Research questions

• RQ1 Why do likelihood-based anomaly detectors using deep generative models often misrank outliers when trained on natural images?
• RQ2 Can a hierarchy of distributions and a hierarchy of features explain and mitigate model bias in anomaly detection?
• RQ3 Do log-likelihood ratios between a specific-in-distribution model and a general-distribution model improve unsupervised anomaly detection across datasets?
• RQ4 Does an outlier loss further enhance anomaly detection performance by steering the model away from high-likelihood general-distribution samples?
• RQ5 How does the final-scale (high-level) likelihood contribution behave across models and datasets, and can it serve as a robust anomaly score?

Key findings

• Low-level, domain-prior features dominate likelihoods in convolutional Glow models, hindering high-level anomaly detection.
• Likelihoods from fully-connected Glow correlations differ, indicating model bias toward low-level features.
• A hierarchy of distributions enables effective log-likelihood-ratio anomaly scoring, often outperforming raw likelihoods.
• An outlier loss using general-distribution samples consistently improves unsupervised anomaly detection.
• Using the last-scale likelihood contribution (z_3) provides strong anomaly signals when general-distribution data is unavailable.
• In supervised settings, class-conditioned models plus outlier samples yield competitive results, approaching state-of-the-art classifier-based methods.