[Paper Review] Adaptive Memory Networks with Self-supervised Learning for Unsupervised Anomaly Detection

Yuxin Zhang, Jindong Wang+3 more|arXiv (Cornell University)|Jan 3, 2022

Anomaly Detection Techniques and Applications5 citations

TL;DR

This paper proposes Adaptive Memory Networks with Self-supervised Learning (AMSL), a novel unsupervised anomaly detection method for multivariate time series that enhances generalization by combining self-supervised learning to capture diverse normal patterns and an adaptive memory fusion module to learn rich, representative features. On the large-scale CAP sleep stage dataset (900M samples), AMSL outperforms the next-best method by over 4% in both F1 and accuracy, demonstrating superior performance, robustness, and efficiency.

ABSTRACT

Unsupervised anomaly detection aims to build models to effectively detect unseen anomalies by only training on the normal data. Although previous reconstruction-based methods have made fruitful progress, their generalization ability is limited due to two critical challenges. First, the training dataset only contains normal patterns, which limits the model generalization ability. Second, the feature representations learned by existing models often lack representativeness which hampers the ability to preserve the diversity of normal patterns. In this paper, we propose a novel approach called Adaptive Memory Network with Self-supervised Learning (AMSL) to address these challenges and enhance the generalization ability in unsupervised anomaly detection. Based on the convolutional autoencoder structure, AMSL incorporates a self-supervised learning module to learn general normal patterns and an adaptive memory fusion module to learn rich feature representations. Experiments on four public multivariate time series datasets demonstrate that AMSL significantly improves the performance compared to other state-of-the-art methods. Specifically, on the largest CAP sleep stage detection dataset with 900 million samples, AMSL outperforms the second-best baseline by extbf{4}\%+ in both accuracy and F1 score. Apart from the enhanced generalization ability, AMSL is also more robust against input noise.

Motivation & Objective

• To address the limited generalization of unsupervised anomaly detection models trained only on normal data.
• To overcome the challenge of insufficiently representative feature representations that fail to capture the diversity of normal patterns.
• To improve model robustness against unseen anomalies and input noise.
• To develop a method that generalizes effectively to large-scale, real-world multivariate time series data.

Proposed method

• Integrates a self-supervised learning module within a convolutional autoencoder framework to learn generalizable normal patterns from unlabeled normal data.
• Employs a global memory module to capture common, shared features across the time series and a local memory module to preserve instance-specific, detailed patterns.
• Uses an adaptive memory fusion module that dynamically combines global and local representations based on input context, improving feature representativeness.
• Leverages contrastive learning with data augmentations (e.g., noise, scaling, permutation, reversal) to improve self-supervised representation learning.
• Optimizes the model end-to-end using a multi-component loss function that balances reconstruction loss, self-supervised contrastive loss, and regularization.
• Applies a threshold based on the 99th percentile of reconstruction errors to detect anomalies, ensuring robustness to distribution shifts.

Figure 1: Illustration of AMSL. Even with similar inputs ((a) and (b)), existing methods can overfit to the anomalies; On the other hand, when the inputs are diverse ((a) and (c)), our method remains accurate while existing methods can fail. $Axis$ represents three dimensional signals on dataset.

Experimental results

Research questions

• RQ1 Can self-supervised learning improve the generalization of unsupervised anomaly detection models when only normal data is available?
• RQ2 How can memory networks be designed to preserve both global commonality and local diversity in normal time series patterns?
• RQ3 Does adaptive fusion of global and local memory representations enhance detection performance on unseen anomalies?
• RQ4 How does the proposed method perform under high-dimensional, large-scale time series data with limited labeled anomalies?
• RQ5 Can the model maintain high performance and robustness under input noise and distribution shifts?

Key findings

• On the CAP sleep stage detection dataset with 900 million samples, AMSL achieves a 4.1% improvement in F1 score and 4.2% in accuracy over the second-best baseline method.
• AMSL achieves the highest F1 score (0.9750) and accuracy (0.9770) when using the 99th percentile as the anomaly threshold, outperforming thresholds at the 90th and 95th percentiles.
• The model converges stably and quickly, with reconstruction and self-supervised losses showing consistent improvement during training.
• AMSL maintains strong performance even when reducing the number of self-supervised data augmentations (e.g., discarding 'Noise' or 'Scale' transformations), with F1 scores above 0.90 even with only three transformations.
• AMSL has fewer parameters (1.5M) and a smaller model size (5.7MB) than most baselines, while achieving superior performance, indicating high efficiency.

Figure 2: The structure of the proposed AMSL. It consists of four components: self-supervised learning, global memory, local memory and adaptive fusion. The notation “ $\times R$ ” denotes $R$ copies where each one corresponds to one transformation.