[Paper Review] A Comparative Study on Unsupervised Anomaly Detection for Time Series: Experiments and Analysis

Yan Zhao, Liwei Deng+8 more|arXiv (Cornell University)|Sep 10, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper presents a comprehensive comparative study of unsupervised time series anomaly detection methods, introducing unified taxonomies for data, methods, and evaluation strategies. It evaluates state-of-the-art traditional and deep learning techniques across nine public datasets using standardized implementations and metrics, offering evidence-based guidelines for method selection and identifying key performance trends across diverse application settings.

ABSTRACT

The continued digitization of societal processes translates into a proliferation of time series data that cover applications such as fraud detection, intrusion detection, and energy management, where anomaly detection is often essential to enable reliability and safety. Many recent studies target anomaly detection for time series data. Indeed, area of time series anomaly detection is characterized by diverse data, methods, and evaluation strategies, and comparisons in existing studies consider only part of this diversity, which makes it difficult to select the best method for a particular problem setting. To address this shortcoming, we introduce taxonomies for data, methods, and evaluation strategies, provide a comprehensive overview of unsupervised time series anomaly detection using the taxonomies, and systematically evaluate and compare state-of-the-art traditional as well as deep learning techniques. In the empirical study using nine publicly available datasets, we apply the most commonly-used performance evaluation metrics to typical methods under a fair implementation standard. Based on the structuring offered by the taxonomies, we report on empirical studies and provide guidelines, in the form of comparative tables, for choosing the methods most suitable for particular application settings. Finally, we propose research directions for this dynamic field.

Motivation & Objective

• To address the lack of systematic, fair comparisons in unsupervised time series anomaly detection due to fragmented data, method, and evaluation diversity.
• To develop standardized taxonomies for time series data, anomaly detection methods, and evaluation strategies to enable consistent benchmarking.
• To conduct a systematic empirical evaluation of state-of-the-art traditional and deep learning methods under a unified implementation framework.
• To provide data-driven, comparative tables and practical guidelines for selecting optimal methods based on specific application settings.
• To identify open challenges and propose future research directions in unsupervised time series anomaly detection.

Proposed method

• The authors design a multi-dimensional taxonomy for time series data, covering characteristics such as domain, temporal resolution, and noise level.
• They categorize anomaly detection methods into traditional (e.g., statistical, reconstruction-based) and deep learning-based (e.g., autoencoders, LSTM-based models) approaches.
• A standardized evaluation protocol is established, ensuring all methods are implemented consistently across all datasets using common hyperparameters and training procedures.
• Performance is measured using widely adopted metrics such as AUC-ROC, F1-score, and precision-recall, applied uniformly across all methods and datasets.
• The study uses nine publicly available time series datasets spanning domains like energy, finance, and system monitoring to ensure broad applicability.
• Comparative analysis is conducted using structured tables and visualizations to highlight method performance across different data and problem characteristics.

Experimental results

Research questions

• RQ1 Which unsupervised anomaly detection methods perform best across diverse time series datasets under standardized evaluation?
• RQ2 How do method performance and robustness vary across different data characteristics such as noise level, length, and domain?
• RQ3 What are the relative strengths and weaknesses of traditional versus deep learning-based anomaly detection techniques in real-world settings?
• RQ4 How do different evaluation metrics correlate with each other, and which are most reliable for method selection?
• RQ5 What practical guidelines can be derived for selecting the most suitable method based on specific application constraints and data properties?

Key findings

• Deep learning-based methods, particularly autoencoder variants, consistently outperform traditional statistical methods on complex, high-dimensional time series data.
• The performance of all methods is highly sensitive to data characteristics such as noise level and temporal resolution, with significant drops in AUC-ROC observed in noisy or irregularly sampled series.
• Reconstruction-based models like Variational Autoencoders (VAEs) and Denoising Autoencoders (DAEs) show strong generalization across diverse datasets, especially when data is sparse or contains outliers.
• Traditional methods such as Isolation Forest and One-Class SVM perform competitively on low-dimensional, clean time series but degrade significantly on high-dimensional or long-sequence data.
• Evaluation metrics such as AUC-ROC and F1-score often yield conflicting rankings, highlighting the importance of selecting metrics aligned with application-specific needs.
• The study identifies a lack of reproducibility in prior work due to inconsistent implementation and hyperparameter choices, underscoring the need for standardized benchmarks.