[Paper Review] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

Jiehui Xu, Haixu Wu+2 more|arXiv (Cornell University)|Oct 6, 2021

Anomaly Detection Techniques and Applications30 references238 citations

TL;DR

This paper proposes the Anomaly Transformer, a novel unsupervised time series anomaly detection model that leverages self-attention mechanisms to compute an Association Discrepancy—a measure of how differently anomalous points associate with the full time series compared to normal points. By introducing a two-branch Anomaly-Attention mechanism and a minimax strategy to amplify the distinguishability between normal and anomalous associations, the model achieves state-of-the-art performance across six benchmarks in service monitoring, space/earth exploration, and water treatment.

ABSTRACT

Anomaly detection in time series data is a critical task in various domains, including finance, healthcare, cybersecurity and industry. Traditional methods, such as time series decomposition, clustering, and density estimation, have provided robust solutions for identifying anomalies that exhibit distinct patterns or significant deviations from normal data distributions. Recent advancements in machine learning and deep learning have further enhanced these capabilities. This paper introduces a novel method for anomaly detection that combines the strengths of autoencoders and recurrent neural networks (RNNs) with an reconstruction error feedback mechanism based on Mean Squared Error. We compare our method against classical techniques and recent approaches like OmniAnomaly, which leverages stochastic recurrent neural networks, and the Anomaly Transformer, which introduces association discrepancy to capture long-range dependencies and DCDetector using contrastive representation learning with multi-scale dual attention. Experimental results demonstrate that our method achieves superior overall performance in terms of precision, recall, and F1 score. The source code is available at http://github.com/mribrahim/AE-FAR

Motivation & Objective

• To address the challenge of unsupervised time series anomaly detection, where anomalies are rare and hard to distinguish from normal data.
• To develop a criterion that effectively captures the inherent differences in temporal association patterns between normal and anomalous time points.
• To improve detection performance by modeling both prior-association (adjacent-concentration bias) and series-association (global attention) simultaneously.
• To enhance model robustness and distinguishability through a novel minimax optimization strategy on the association discrepancy.

Proposed method

• Proposes the Anomaly-Attention mechanism with a two-branch structure: one branch models the prior-association using a learnable Gaussian kernel with adjacent-concentration inductive bias, and the other learns the series-association from raw time series via standard self-attention.
• Defines Association Discrepancy as the distance between the prior-association and series-association for each time point, capturing the deviation in how points relate to the whole series.
• Introduces a minimax strategy that optimizes the prior-association to constrain the series-association, thereby amplifying the distinguishability of the association discrepancy between normal and anomalous points.
• Uses reconstruction loss to preserve representation quality while the minimax objective enhances the discriminative power of the association discrepancy for anomaly detection.
• Applies the association discrepancy as a new detection criterion, replacing or complementing traditional reconstruction error.
• Trains the model end-to-end in an unsupervised manner using only raw time series data, without requiring labeled anomalies.

Experimental results

Research questions

• RQ1 Can the association discrepancy between prior-association (adjacent-concentration) and series-association (global attention) serve as a reliable and distinguishable anomaly criterion?
• RQ2 How can the inherent adjacent-concentration bias of anomalies be modeled and exploited to improve detection performance?
• RQ3 Can a minimax strategy effectively amplify the normal-abnormal distinction in association discrepancy to enhance detection robustness?
• RQ4 Does the proposed Anomaly Transformer outperform existing state-of-the-art methods across diverse real-world time series applications?

Key findings

• The Anomaly Transformer achieves state-of-the-art performance on six unsupervised time series anomaly detection benchmarks across three real-world applications: service monitoring, space & earth exploration, and water treatment.
• The minimax strategy significantly improves the contrast between abnormal and normal time points, increasing the association discrepancy contrast from 1.15 (reconstruction-only) to 2.39 on the SMD dataset.
• The association-based criterion outperforms reconstruction-based criteria, especially in detecting point-contextual and pattern-seasonal anomalies, reducing false positives.
• The learned scale parameter σ in the prior-attention branch shows smaller values for anomalies, confirming the adjacent-concentration inductive bias is effectively captured.
• Visualization confirms that the association discrepancy criterion produces more consistent and distinguishable scores for anomalies compared to reconstruction error.
• Ablation studies demonstrate that both the Anomaly-Attention mechanism and the minimax strategy are essential for optimal performance, with ablation on either component leading to significant performance drops.