[Paper Review] Deep Variational Semi-Supervised Novelty Detection

Tal Daniel, Thanard Kurutach+1 more|arXiv (Cornell University)|Nov 12, 2019

Anomaly Detection Techniques and Applications47 references4 citations

TL;DR

This paper proposes two principled variational inference methods—Dual-Prior VAE and CUBO-regularized VAE—for semi-supervised anomaly detection using variational autoencoders (VAEs), which explicitly model outlier data during training. By encouraging the encoder to separate normal and anomalous latent representations, the methods achieve state-of-the-art or competitive performance across diverse datasets, including images, medical, and astronomical data, even with only 1% labeled anomalies.

ABSTRACT

In anomaly detection (AD), one seeks to identify whether a test sample is abnormal, given a data set of normal samples. A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs), for unsupervised learning of the normal data distribution. In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies. In this work, we propose two variational methods for training VAEs for SSAD. The intuitive idea in both methods is to train the encoder to `separate' between latent vectors for normal and outlier data. We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms. Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, can be combined with any VAE model architecture, and are naturally compatible with ensembling. When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

Motivation & Objective

• To address the limitation of unsupervised VAEs in detecting out-of-distribution samples by incorporating a small set of labeled anomalies.
• To develop a principled probabilistic framework for training VAEs that explicitly accounts for negative (outlier) examples during training.
• To improve generalization in anomaly detection by modifying the VAE objective to separate normal and anomalous latent representations.
• To demonstrate the flexibility and effectiveness of the proposed methods across diverse data modalities, including images, medical data, and text.
• To enable principled use of model ensembling in VAE-based anomaly detection, enhancing robustness and performance.

Proposed method

• Proposes a Dual-Prior VAE (DP-VAE) that uses separate prior distributions for normal and outlier data, encouraging the encoder to map outliers to a distinct latent space.
• Introduces a CUBO-regularized VAE that combines the standard ELBO with the chi-upper bound (CUBO) to jointly maximize likelihood on normal data and minimize it on outliers.
• Derives both methods from principled probabilistic formulations of the SSAD problem, ensuring theoretical grounding.
• Applies the methods to any VAE architecture without architectural constraints, enabling compatibility with existing models.
• Supports ensembling of multiple VAEs trained with the proposed objectives, improving detection robustness.
• Demonstrates generalization by applying the method to non-AD tasks, such as motion planning and sentiment analysis, by incorporating negative samples during training.

Experimental results

Research questions

• RQ1 Can a principled probabilistic formulation improve VAE-based anomaly detection when a small number of labeled outliers are available?
• RQ2 How can the VAE training objective be modified to explicitly separate normal and anomalous latent representations?
• RQ3 To what extent does incorporating even 1% of labeled anomalies improve detection performance across diverse data types?
• RQ4 Can the proposed methods be effectively combined with model ensembling to further enhance detection performance?
• RQ5 Can the approach be generalized beyond anomaly detection to other generative modeling tasks where negative examples are available?

Key findings

• The proposed DP-VAE and CUBO-regularized VAE methods achieve state-of-the-art or competitive AUROC scores on the comprehensive SSAD benchmark, including on CIFAR-10, satellite, and cardio datasets.
• On CIFAR-10 with 1% anomalies, the method achieves an AUROC of 74.5 ± 8.4 using a frozen decoder and ensemble of 5 models, outperforming prior methods.
• Abllation studies show that using a shared encoder for normal and outlier data is critical, while freezing the decoder provides minimal benefit with limited outlier data.
• The method significantly improves sample quality in motion planning: DP-VAE generates fewer collision-prone configurations compared to standard CVAE, as shown in Figure 3.
• The approach generalizes beyond anomaly detection, successfully enhancing a conditional VAE for robot motion planning by incorporating obstacle-boundary samples as negative examples.
• Even a small fraction of labeled anomalies (1%) leads to marked improvement in outlier detection, demonstrating the method's data efficiency.