[Paper Review] PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection

Ronghui Xu, Hao Miao+3 more|arXiv (Cornell University)|Jun 4, 2024

Anomaly Detection Techniques and Applications4 citations

TL;DR

PeFAD proposes a parameter-efficient federated learning framework for time series anomaly detection that leverages pre-trained language models (PLM) for robust representation learning while minimizing communication and computation costs through fine-tuning of small-scale adapter parameters. It introduces an anomaly-driven mask selection strategy and a privacy-preserving synthetic dataset distillation mechanism to improve detection accuracy and handle data heterogeneity, achieving up to 28.74% performance gain over SOTA baselines on real-world datasets.

ABSTRACT

With the proliferation of mobile sensing techniques, huge amounts of time series data are generated and accumulated in various domains, fueling plenty of real-world applications. In this setting, time series anomaly detection is practically important. It endeavors to identify deviant samples from the normal sample distribution in time series. Existing approaches generally assume that all the time series is available at a central location. However, we are witnessing the decentralized collection of time series due to the deployment of various edge devices. To bridge the gap between the decentralized time series data and the centralized anomaly detection algorithms, we propose a Parameter-efficient Federated Anomaly Detection framework named PeFAD with the increasing privacy concerns. PeFAD for the first time employs the pre-trained language model (PLM) as the body of the client's local model, which can benefit from its cross-modality knowledge transfer capability. To reduce the communication overhead and local model adaptation cost, we propose a parameter-efficient federated training module such that clients only need to fine-tune small-scale parameters and transmit them to the server for update. PeFAD utilizes a novel anomaly-driven mask selection strategy to mitigate the impact of neglected anomalies during training. A knowledge distillation operation on a synthetic privacy-preserving dataset that is shared by all the clients is also proposed to address the data heterogeneity issue across clients. We conduct extensive evaluations on four real datasets, where PeFAD outperforms existing state-of-the-art baselines by up to 28.74%.

Motivation & Objective

• To address the challenge of decentralized time series anomaly detection under strict privacy constraints, where data cannot be centrally aggregated.
• To overcome data scarcity and poor representation learning in federated settings, especially for rare anomalous patterns.
• To reduce communication and computational overhead in federated training while maintaining model performance.
• To improve anomaly detection robustness by mitigating the impact of anomalies during model pre-training and fine-tuning.
• To address client data heterogeneity through a privacy-preserving shared synthetic dataset generation mechanism.

Proposed method

• Employs a pre-trained language model (PLM) as the backbone for local client models to leverage cross-modality knowledge and improve representation learning.
• Introduces a parameter-efficient federated training module where only small adapter parameters are fine-tuned and transmitted, reducing communication and computation costs.
• Proposes an anomaly-driven mask selection (ADMS) strategy that dynamically selects masking patterns based on anomaly likelihood to enhance reconstruction robustness.
• Designs a privacy-preserving dataset synthesis mechanism using mutual information and Wasserstein distance constraints to generate consistent, private synthetic data shared across clients.
• Applies knowledge distillation on the shared synthetic dataset to align client models and improve generalization across heterogeneous data distributions.
• Uses a dual-contrastive learning objective to enhance feature discrimination and model robustness during federated training.

Experimental results

Research questions

• RQ1 Can a pre-trained language model effectively improve representation learning for time series anomaly detection in a federated setting with limited local data?
• RQ2 How can parameter-efficient fine-tuning reduce communication and computational overhead in federated time series anomaly detection?
• RQ3 To what extent does anomaly-driven mask selection improve detection performance by focusing on dynamic and anomalous regions?
• RQ4 Can a privacy-preserving synthetic dataset mitigate the impact of non-IID data across clients in federated anomaly detection?
• RQ5 How does knowledge distillation on shared synthetic data enhance model generalization and convergence in heterogeneous federated settings?

Key findings

• PeFAD achieves up to 28.74% improvement in F1-score over state-of-the-art baselines on four real-world time series datasets.
• The use of pre-trained language models significantly enhances representation learning, especially in low-data regimes, improving detection performance on sparse anomalous patterns.
• The anomaly-driven mask selection (ADMS) strategy increases model sensitivity to anomalies by focusing on dynamic and changing regions in time series.
• The parameter-efficient fine-tuning module reduces communication overhead by transmitting only small adapter parameters, maintaining high model accuracy.
• The privacy-preserving synthetic dataset generation mechanism, combining mutual information and Wasserstein distance constraints, produces high-quality, private data that improves client model alignment and global model stability.
• Ablation studies confirm that each component—PLM, ADMS, and PPDS—contributes significantly to the overall performance, with the PLM backbone providing the strongest gain in representation learning.