[Paper Review] PatchAD: A Lightweight Patch-based MLP-Mixer for Time Series Anomaly Detection

Zhijie Zhong, Zhiwen Yu+3 more|arXiv (Cornell University)|Jan 18, 2024

Anomaly Detection Techniques and Applications4 citations

TL;DR

PatchAD proposes a lightweight, multi-scale patch-based MLP-Mixer architecture for time series anomaly detection using contrastive learning to distinguish normal from anomalous patterns. By leveraging non-overlapping temporal patches and a dual project constraint, it achieves state-of-the-art performance with high efficiency and robustness across eight benchmark datasets.

ABSTRACT

Time series anomaly detection is a pivotal task in data analysis, yet it poses the challenge of discerning normal and abnormal patterns in label-deficient scenarios. While prior studies have largely employed reconstruction-based approaches, which limit the models' representational capacities. Moreover, existing deep learning-based methods are not sufficiently lightweight. Addressing these issues, we present PatchAD, our novel, highly efficient multiscale patch-based MLP-Mixer architecture that utilizes contrastive learning for representation extraction and anomaly detection. With its four distinct MLP Mixers and innovative dual project constraint module, PatchAD mitigates potential model degradation and offers a lightweight solution, requiring only $0.403M$ parameters. Its efficacy is demonstrated by state-of-the-art results across $8$ datasets sourced from different application scenarios, outperforming over $30$ comparative algorithms. PatchAD significantly improves the classical F1 score by 6.84%, the Aff-F1 score by 4.27%, and the V-ROC by 2.49%. Simultaneously, an in-depth analysis of the mechanisms underlying PatchAD has been conducted from both theoretical and experimental perspectives, validating the design motivations of the model. The code is publicly available at https://github.com/EmorZz1G/PatchAD.

Motivation & Objective

• To address the limitations of reconstruction-based methods in distinguishing normal and anomalous patterns in unlabeled time series data.
• To design a lightweight deep learning framework that maintains high performance while reducing computational complexity.
• To improve representational learning in time series anomaly detection by modeling multi-scale temporal patches and inter-channel dependencies.
• To mitigate model degradation in contrastive learning through a novel dual project constraint module.
• To enable effective anomaly detection across diverse multivariate time series datasets with minimal architectural overhead.

Proposed method

• PatchAD employs a multi-scale patching strategy that divides input time series into non-overlapping temporal patches to capture richer semantic features beyond point-level representations.
• It uses four distinct MLP-Mixer encoders, each processing different patch sizes to model multi-scale temporal dynamics and relationships.
• The model applies contrastive learning by creating two augmented views of the same patch sequence, encouraging consistent representations for normal samples while disrupting those for anomalies.
• A dual project constraint module is introduced to stabilize training and prevent representation collapse in contrastive learning by regularizing the projection heads.
• Inter-patch and intra-patch relationships are modeled through MLP-based mixing mechanisms, avoiding attention mechanisms to ensure lightweight inference.
• Anomaly scores are computed based on the contrastive loss and reconstruction error, with a learned threshold used to classify anomalies.

Experimental results

Research questions

• RQ1 Can a purely MLP-based architecture outperform attention-based models in time series anomaly detection while maintaining lower computational cost?
• RQ2 How effective is multi-scale patching in improving the model’s ability to detect diverse anomaly types across different time series patterns?
• RQ3 To what extent does the dual project constraint mitigate representation collapse in contrastive learning for time series data?
• RQ4 How does PatchAD perform across diverse real-world multivariate time series datasets compared to state-of-the-art methods?
• RQ5 What hyperparameter settings (e.g., window size, patch size, model dimension) yield optimal trade-offs between performance and inference speed?

Key findings

• PatchAD achieves state-of-the-art performance on eight benchmark datasets, with an F1 score of 95.02 on MSL, 96.55 on SMAP, and 92.43 on WADI.
• The ablation study confirms that removing the dual project constraint reduces F1 by 1.96 points on MSL, demonstrating its critical role in preventing model degradation.
• The model maintains high efficiency, being 1.45x faster than DCdetector at a sliding step of 1 on the SMD dataset.
• Performance remains stable across varying window sizes (60–210), with optimal results observed at window size 105, indicating strong robustness to input length variation.
• The model achieves 100% recall on the WADI dataset when using the full configuration, highlighting its strong detection capability for rare or complex anomalies.
• Sensitivity analysis shows that setting the projection constraint between 0.1 and 0.4, and the anomaly threshold between 0.8 and 1.0, yields the best performance across datasets.