[Paper Review] TiSAT: Time Series Anomaly Transformer

Keval Doshi, Shatha Abudalou+1 more|arXiv (Cornell University)|Mar 10, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes TiSAT, a self-attention-based transformer model for unsupervised time series anomaly detection that addresses flawed evaluation metrics by introducing a novel SPD metric measuring detection timeliness and precision. TiSAT outperforms state-of-the-art methods on benchmark datasets using a closed-form threshold derived from asymptotic false alarm rate analysis, demonstrating superior performance when evaluated with the corrected metric.

ABSTRACT

While anomaly detection in time series has been an active area of research for several years, most recent approaches employ an inadequate evaluation criterion leading to an inflated F1 score. We show that a rudimentary Random Guess method can outperform state-of-the-art detectors in terms of this popular but faulty evaluation criterion. In this work, we propose a proper evaluation metric that measures the timeliness and precision of detecting sequential anomalies. Moreover, most existing approaches are unable to capture temporal features from long sequences. Self-attention based approaches, such as transformers, have been demonstrated to be particularly efficient in capturing long-range dependencies while being computationally efficient during training and inference. We also propose an efficient transformer approach for anomaly detection in time series and extensively evaluate our proposed approach on several popular benchmark datasets.

Motivation & Objective

• To identify and rectify the flawed evaluation criterion in existing time series anomaly detection methods, which artificially inflates F1 scores by over-penalizing false negatives and over-amplifying true positives in sequential anomalies.
• To propose a new evaluation metric, SPD (Sequential Precision and Delay), that measures both the timeliness and precision of anomaly detection in sequential patterns.
• To develop an efficient transformer-based model, TiSAT, capable of capturing long-range temporal dependencies in time series data while maintaining computational efficiency.
• To provide a theoretical foundation for setting detection thresholds by deriving a closed-form expression for the false alarm rate using extreme value theory and the Lambert-W function.

Proposed method

• TiSAT employs a two-encoder, one-decoder transformer architecture with a sliding window input of size 100 to process sequential time series data.
• The model uses a reconstruction-based loss to detect anomalies by measuring the reconstruction error of the input sequence.
• A key innovation is the derivation of a closed-form threshold using extreme value theory, where the threshold h is computed as h = -log(FAR)/ω₀, with ω₀ derived from data dimensionality and statistical properties.
• The false alarm rate is analytically bounded using the inverse of the expected false alarm period, validated empirically on five benchmark datasets.
• The model is trained using Adam optimizer with a fixed learning rate of 1e-4 over 4 epochs and min-max normalization applied to all datasets.
• The proposed SPD metric evaluates detection performance by measuring the delay in alarm triggering and the precision of anomaly sequence detection, avoiding the bias of traditional instance-based metrics.

Experimental results

Research questions

• RQ1 Can a standard instance-based evaluation metric that amplifies true positives in sequential anomalies lead to misleadingly high F1 scores, even when the model performs poorly in practice?
• RQ2 Does a simple random guess strategy, which randomly raises alarms with a low probability, outperform state-of-the-art anomaly detection models under the standard evaluation protocol?
• RQ3 Can a novel evaluation metric—SPD—better reflect real-world performance by measuring both detection timeliness and precision in sequential anomaly detection?
• RQ4 Can a transformer-based architecture effectively model long-range temporal dependencies in time series while maintaining computational efficiency during training and inference?
• RQ5 Is it possible to derive a closed-form analytical expression for the detection threshold that ensures a desired false alarm rate in unsupervised anomaly detection?

Key findings

• The Random Guess method with a 1% alarm probability achieves an F1 score of 93.39 on SMD and 98.83 on PSM, outperforming state-of-the-art models under the standard adjusted instance-based evaluation, exposing its fundamental flaw.
• TiSAT achieves an SPD score of 52.70 on SMD, 33.40 on MSL, 29.35 on SMAP, 46.23 on SWaT, and 38.58 on PSM, significantly outperforming classical methods like OCSVM, IsolationForest, and LSTM.
• The proposed SPD metric reveals that models like Anomaly Transformer and InterFusion, which dominate in F1 scores, perform poorly in terms of detection timeliness and precision.
• The theoretical threshold derived from the asymptotic false alarm rate analysis closely matches empirical results, validating the closed-form expression h = -log(FAR)/ω₀ across all datasets.
• The model’s reconstruction-based training with attention mechanisms enables effective capture of long-range dependencies, leading to superior detection performance on diverse, real-world time series data.
• Empirical validation confirms that the lower bound on the false alarm period derived from Theorem 1 closely matches actual observed false alarm periods, supporting the robustness of the theoretical framework.