[論文レビュー] TSLANet: Rethinking Transformers for Time Series Representation Learning

Emadeldeen Eldele, Mohamed Ragab+3 more|arXiv (Cornell University)|Apr 12, 2024

Time Series Analysis and Forecasting被引用数 22

ひとこと要約

TSLANetは、FFTベースの処理と適応高周波閾値処理を備えたAdaptive Spectral Block、Interactive Convolution Block、自己教師付き事前学習を用いて、多様なタスクに対応する普遍的で軽量な時系列モデルを提示します。

ABSTRACT

Time series data, characterized by its intrinsic long and short-range dependencies, poses a unique challenge across analytical applications. While Transformer-based models excel at capturing long-range dependencies, they face limitations in noise sensitivity, computational efficiency, and overfitting with smaller datasets. In response, we introduce a novel Time Series Lightweight Adaptive Network (TSLANet), as a universal convolutional model for diverse time series tasks. Specifically, we propose an Adaptive Spectral Block, harnessing Fourier analysis to enhance feature representation and to capture both long-term and short-term interactions while mitigating noise via adaptive thresholding. Additionally, we introduce an Interactive Convolution Block and leverage self-supervised learning to refine the capacity of TSLANet for decoding complex temporal patterns and improve its robustness on different datasets. Our comprehensive experiments demonstrate that TSLANet outperforms state-of-the-art models in various tasks spanning classification, forecasting, and anomaly detection, showcasing its resilience and adaptability across a spectrum of noise levels and data sizes. The code is available at https://github.com/emadeldeen24/TSLANet.

研究の動機と目的

• Introduce a universal lightweight network for diverse time series tasks.
• Capture both long- and short-range dependencies with convolutional operations.
• Reduce noise sensitivity and improve efficiency compared to Transformers.
• Enhance feature representation via adaptive spectral processing and interactive convolutions.
• Leverage self-supervised pretraining to boost robustness across datasets.

提案手法

• Replace self-attention with an Adaptive Spectral Block that performs Fourier-domain processing and adaptive high-frequency thresholding.
• Use global and local learnable filters in the spectral domain to create an integrated spectral representation.
• Introduce an Interactive Convolution Block with multi-kernel convolutions whose outputs modulate each other.
• Split input series into patches with learnable positional embeddings to form embeddings for processing.
• Apply inverse FFT to return to time domain after spectral processing.
• Incorporate self-supervised pretraining via a masked autoencoder paradigm focusing on patch-level reconstruction.

Figure 1 : A comparison between CNN and Transformer-based architectures for classification and forecasting tasks. Classification results are the average over 10 UEA datasets ( Wu et al. , 2023 ) , while forecasting results are the average MSE results on lengths {96, 192, 336, 720}.

実験結果

リサーチクエスチョン

• RQ1 Can a convolutional-based architecture with adaptive spectral processing match or exceed Transformer-based models in time series classification, forecasting, and anomaly detection?
• RQ2 How does adaptive frequency-domain filtering influence noise robustness and long-/short-range dependency modeling?
• RQ3 What is the impact of the Interactive Convolution Block on capturing multi-scale temporal patterns?
• RQ4 Does self-supervised pretraining improve performance and robustness across diverse time series datasets?

主な発見

• TSLANet achieves superior performance to several state-of-the-art baselines across classification, forecasting, and anomaly detection tasks.
• The Adaptive Spectral Block (ASB) with adaptive high-frequency thresholding reduces noise while preserving relevant spectral information.
• The Interactive Convolution Block enhances feature extraction by enabling interactions between multi-scale convolutions.
• Self-supervised pretraining further improves representation quality and robustness across datasets.
• TSLANet offers competitive accuracy with lower FLOPs and parameter counts compared to some Transformer-based models.
• Ablation studies show substantial performance drops when removing ASB or ICB, validating their contributions.

Figure 2 : The structure of our proposed TSLANet . The input time series is split into patches, and positional embeddings are added. Next, the output embeddings pass through TSLANet layers, where each layer consists of two main components. The first is the Adaptive Spectral Block, which leverages fr