[Paper Review] Metric Learning for Novelty and Anomaly Detection

Marc Masana, Idoia Ruiz+3 more|arXiv (Cornell University)|Aug 16, 2018

Anomaly Detection Techniques and ApplicationsComputer Science23 references51 citations

TL;DR

The paper proposes a metric-learning approach to out-of-distribution detection, distinguishing novelty from anomaly, and shows competitive results vs cross-entropy methods across several benchmarks, including traffic sign recognition.

ABSTRACT

When neural networks process images which do not resemble the distribution seen during training, so called out-of-distribution images, they often make wrong predictions, and do so too confidently. The capability to detect out-of-distribution images is therefore crucial for many real-world applications. We divide out-of-distribution detection between novelty detection ---images of classes which are not in the training set but are related to those---, and anomaly detection ---images with classes which are unrelated to the training set. By related we mean they contain the same type of objects, like digits in MNIST and SVHN. Most existing work has focused on anomaly detection, and has addressed this problem considering networks trained with the cross-entropy loss. Differently from them, we propose to use metric learning which does not have the drawback of the softmax layer (inherent to cross-entropy methods), which forces the network to divide its prediction power over the learned classes. We perform extensive experiments and evaluate both novelty and anomaly detection, even in a relevant application such as traffic sign recognition, obtaining comparable or better results than previous works.

Motivation & Objective

• Motivate the importance of detecting out-of-distribution inputs in neural networks.
• Distinguish novelty (related unseen classes) from anomaly (unrelated data) in OOD detection.
• Propose a metric-learning framework that avoids softmax-based confidence pitfalls of cross-entropy.
• Demonstrate that embedding-based approaches can yield strong novelty and anomaly detection results.
• Apply the method to real-world traffic sign recognition to validate practical performance.

Proposed method

• Introduce an embedding where Euclidean distance measures out-of-distribution likelihood.
• Use metric learning losses (contrastive loss) to pull same-class samples together and push different-class samples apart.
• Extend the loss to incorporate out-of-distribution mining (ODM) by including seen/unseen out-of-distribution data during training.
• Employ a single-branch network to efficiently mine pairs for the embedding without a Siamese architecture.
• Compare metric learning (ML) and out-of-distribution mining (ODM) against cross-entropy baselines and state-of-the-art methods (CC-AG, ODIN).

Experimental results

Research questions

• RQ1 Can a metric-learning embedding improve novelty and anomaly detection compared to softmax-based classifiers?
• RQ2 How do novelty and anomaly out-of-distribution samples differ in embedding space, and how does ODM influence this separation?
• RQ3 Does incorporating seen out-of-distribution data during training improve detection for both novelty and anomaly?
• RQ4 Is the proposed ODM approach effective across standard benchmarks (SVHN, CIFAR-10) and real-world tasks (traffic signs)?

Key findings

• Metric learning yields embeddings where in-distribution samples cluster per class and out-of-distribution samples lie away from these clusters.
• Out-of-distribution mining (ODM) further improves separation between in-distribution and out-of-distribution samples, aiding detection.
• Compared to cross-entropy based methods, metric learning approaches show better handling of novelty and anomalies, with improvements in keeping unseen data from being confidently misclassified.
• In traffic sign recognition, ODM achieves competitive novelty and anomaly detection while maintaining reasonable in-distribution classification performance.
• The results suggest a continuum between novelty and anomaly in out-of-distribution data, with anomaly detection being easier than novelty detection.