[Paper Review] DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition

Matej Grcić, Petra Bevandić+1 more|arXiv (Cornell University)|Jul 6, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

DenseHybrid proposes a novel hybrid anomaly detection framework for dense open-set recognition that combines generative likelihood estimation and discriminative data posterior prediction using unnormalized distributions to avoid intractable normalizing constants. By training on both in-distribution and out-of-distribution data, it achieves state-of-the-art performance on dense anomaly detection benchmarks with negligible computational overhead over standard semantic segmentation models.

ABSTRACT

Anomaly detection can be conceived either through generative modelling of regular training data or by discriminating with respect to negative training data. These two approaches exhibit different failure modes. Consequently, hybrid algorithms present an attractive research goal. Unfortunately, dense anomaly detection requires translational equivariance and very large input resolutions. These requirements disqualify all previous hybrid approaches to the best of our knowledge. We therefore design a novel hybrid algorithm based on reinterpreting discriminative logits as a logarithm of the unnormalized joint distribution $\hat{p}(\mathbf{x}, \mathbf{y})$. Our model builds on a shared convolutional representation from which we recover three dense predictions: i) the closed-set class posterior $P(\mathbf{y}|\mathbf{x})$, ii) the dataset posterior $P(d_{in}|\mathbf{x})$, iii) unnormalized data likelihood $\hat{p}(\mathbf{x})$. The latter two predictions are trained both on the standard training data and on a generic negative dataset. We blend these two predictions into a hybrid anomaly score which allows dense open-set recognition on large natural images. We carefully design a custom loss for the data likelihood in order to avoid backpropagation through the untractable normalizing constant $Z(θ)$. Experiments evaluate our contributions on standard dense anomaly detection benchmarks as well as in terms of open-mIoU - a novel metric for dense open-set performance. Our submissions achieve state-of-the-art performance despite neglectable computational overhead over the standard semantic segmentation baseline.

Motivation & Objective

• To address the limitations of purely generative or discriminative approaches in dense anomaly detection, which exhibit distinct failure modes.
• To enable end-to-end training of a hybrid model that operates at pixel-level resolution for large natural images.
• To design a method that avoids backpropagation through the intractable normalizing constant in likelihood estimation.
• To improve robustness in real-world deployment by enabling open-set recognition that detects anomalous pixels while maintaining high accuracy on inliers.
• To introduce a new evaluation metric, open-mIoU, to assess performance in dense open-set recognition scenarios.

Proposed method

• The model uses a shared convolutional backbone to produce three dense predictions: class posterior P(y|x), data posterior P(d_in|x), and unnormalized data likelihood p̂(x).
• The unnormalized likelihood p̂(x) is computed as the sum of exponentiated logits from the classification head, avoiding the need to compute the intractable partition function Z(θ).
• The data posterior P(d_in|x) is learned via a separate branch trained to distinguish in-distribution from out-of-distribution data using a generic negative dataset (e.g., ADE20k).
• A hybrid anomaly score is formed as the ratio (1 - P(d_in|x)) / p̂(x), combining discriminative and generative signals for improved detection.
• A custom loss function is designed to train the likelihood head without requiring gradient computation through Z(θ), enabling stable end-to-end optimization.
• The model is fine-tuned on mixed-content images with pasted negative instances to improve generalization to out-of-distribution patterns.

Experimental results

Research questions

• RQ1 Can a hybrid anomaly detection framework combining generative likelihood and discriminative data posterior improve dense open-set recognition performance?
• RQ2 How can unnormalized likelihood estimation be effectively used in dense anomaly detection without computing the intractable normalizing constant?
• RQ3 Does combining discriminative and generative signals lead to better generalization on out-of-distribution pixels compared to standalone approaches?
• RQ4 Can such a hybrid model be trained end-to-end with minimal computational overhead over standard semantic segmentation models?
• RQ5 How does the proposed open-mIoU metric reflect the trade-off between inlier segmentation accuracy and anomaly detection performance?

Key findings

• DenseHybrid achieves state-of-the-art performance on standard dense anomaly detection benchmarks, with an AP of 46.1% on Fishyscapes and 45.8% on StreetHazards, outperforming prior methods.
• The hybrid anomaly score, combining (1 - P(d_in|x)) / p̂(x), achieves 63.8% AP on Fishyscapes, surpassing both the generative (60.5%) and discriminative (42.9%) components individually.
• The model incurs only 0.1 GFLOPs and 2.8ms additional inference time on two-megapixel images, demonstrating negligible computational overhead.
• On the StreetHazards dataset, DenseHybrid achieves 45.8% open-mIoU, significantly outperforming the max-logit baseline and other SOTA methods.
• The ablation study confirms that the hybrid score consistently improves performance across all metrics, with FPR95 reduced to 4.9% on Fishyscapes, indicating strong anomaly detection capability.
• The proposed method enables real-time inference, with 15.7 frames per second on RTX3090, making it suitable for deployment in autonomous systems.