[Paper Review] Fake It Till You Make It: Towards Accurate Near-Distribution Novelty Detection

Hossein Mirzaei, Mohammadreza Salehi+5 more|arXiv (Cornell University)|May 28, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes a score-based generative model to synthesize near-distribution anomalous data for training novelty detection models, significantly improving performance on subtle, semantically similar anomalies. By fine-tuning on these synthetic near-anomalies, the method reduces the performance gap between standard and near-distribution novelty detection, achieving state-of-the-art AUROC scores across diverse datasets including medical imaging and industrial quality control.

ABSTRACT

We aim for image-based novelty detection. Despite considerable progress, existing models either fail or face a dramatic drop under the so-called "near-distribution" setting, where the differences between normal and anomalous samples are subtle. We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting. Next, we propose to exploit a score-based generative model to produce synthetic near-distribution anomalous data. Our model is then fine-tuned to distinguish such data from the normal samples. We provide a quantitative as well as qualitative evaluation of this strategy, and compare the results with a variety of GAN-based models. Effectiveness of our method for both the near-distribution and standard novelty detection is assessed through extensive experiments on datasets in diverse applications such as medical images, object classification, and quality control. This reveals that our method considerably improves over existing models, and consistently decreases the gap between the near-distribution and standard novelty detection performance. The code repository is available at https://github.com/rohban-lab/FITYMI.

Motivation & Objective

• Address the critical performance drop of state-of-the-art novelty detection models when faced with near-distribution anomalies—subtly different from normal data.
• Identify that existing methods, including SOTA models like PANDA and CSI, suffer up to a 20% AUROC drop in near-ND settings.
• Overcome limitations of GAN-based and outlier exposure methods in near-ND by using a controllable, stable generative approach.
• Demonstrate consistent improvement in both standard and near-distribution novelty detection across diverse domains, including medical imaging and object classification.
• Close the performance gap between standard and near-ND by enabling effective training with synthetic, semantically close anomalous samples.

Proposed method

• Use a score-based generative model (SDE) to synthesize near-distribution anomalous samples by perturbing normal data in latent space with controlled noise.
• Fine-tune a pre-trained feature extractor using both normal data and the generated near-anomalies to improve discrimination capability.
• Leverage k-NN-based anomaly scoring in the feature space for inference, using the fine-tuned model to compute distances between test samples and training features.
• Apply a density and coverage metric to evaluate the quality and diversity of generated samples, ensuring they are semantically plausible and representative.
• Train with a stopping criterion based on FID scores to ensure high-quality generation without overfitting or mode collapse.
• Use a two-stage training protocol: pre-train on normal data, then fine-tune with synthetic near-anomalies to enhance generalization to subtle outliers.

Experimental results

Research questions

• RQ1 Why do state-of-the-art novelty detection models fail in near-distribution settings where anomalies are semantically close to normal data?
• RQ2 Can synthetic data generation improve novelty detection performance when anomalies are only subtly different from normal samples?
• RQ3 How does a score-based generative model compare to GANs or outlier exposure in generating effective near-anomalies for training?
• RQ4 To what extent can fine-tuning on synthetic near-anomalies close the performance gap between standard and near-distribution novelty detection?
• RQ5 Is the proposed method robust to hyperparameters such as k-NN choice and training stopping point?

Key findings

• Existing SOTA novelty detection models, including PANDA and CSI, suffer up to a 20% AUROC drop in near-ND settings compared to standard ND.
• The proposed method achieves an AUROC of 97.0 on the Weather dataset and 91.2 on WBC, significantly outperforming prior methods in near-ND.
• On the MvTecAD dataset, the method achieves an AUROC of 86.4 in near-ND, outperforming the next best method by over 6 percentage points.
• The method is robust to k-NN choice and training stopping point, with FID scores varying by at most 2% across different configurations.
• The method reduces the performance gap between standard and near-ND settings, with consistent improvements across all evaluated datasets and backbone models.
• The use of score-based generative models (SDEs) enables stable, high-quality generation of near-anomalies, avoiding the mode collapse and instability common in GANs.