[论文解读] TSLANet: Rethinking Transformers for Time Series Representation Learning

Emadeldeen Eldele, Mohamed Ragab+3 more|arXiv (Cornell University)|Apr 12, 2024

Time Series Analysis and Forecasting被引用 22

一句话总结

TSLANet 提出了一种通用、轻量化的时间序列卷积模型，使用带自适应谱块的 FFT 基处理和自适应阈值，以及交互卷积块和自监督预训练，以应对多样化任务。

ABSTRACT

Time series data, characterized by its intrinsic long and short-range dependencies, poses a unique challenge across analytical applications. While Transformer-based models excel at capturing long-range dependencies, they face limitations in noise sensitivity, computational efficiency, and overfitting with smaller datasets. In response, we introduce a novel Time Series Lightweight Adaptive Network (TSLANet), as a universal convolutional model for diverse time series tasks. Specifically, we propose an Adaptive Spectral Block, harnessing Fourier analysis to enhance feature representation and to capture both long-term and short-term interactions while mitigating noise via adaptive thresholding. Additionally, we introduce an Interactive Convolution Block and leverage self-supervised learning to refine the capacity of TSLANet for decoding complex temporal patterns and improve its robustness on different datasets. Our comprehensive experiments demonstrate that TSLANet outperforms state-of-the-art models in various tasks spanning classification, forecasting, and anomaly detection, showcasing its resilience and adaptability across a spectrum of noise levels and data sizes. The code is available at https://github.com/emadeldeen24/TSLANet.

研究动机与目标

• 引入一个用于多样化时间序列任务的通用轻量级网络。
• 通过卷积运算捕捉长程和短程依赖。
• 相比 Transformer，降低对噪声的敏感性并提升效率。
• 通过自适应谱处理和交互卷积提升特征表示。
• 利用自监督预训练以提升跨数据集的鲁棒性。

提出的方法

• 用自适应谱块替代自注意力，实现傅里叶域处理和自适应高频阈值化。
• 在谱域中使用全局和局部可学习滤波器，形成集成的谱表示。
• 引入一个具有多核卷积的交互卷积块，其输出相互作用调制。
• 将输入序列拆分为带有可学习位置嵌入的补丁，以形成用于处理的嵌入。
• 在谱处理后应用逆FFT回到时域。
• 通过掩码自编码器范式，将自监督预训练聚焦于补丁级重建。

Figure 1 : A comparison between CNN and Transformer-based architectures for classification and forecasting tasks. Classification results are the average over 10 UEA datasets ( Wu et al. , 2023 ) , while forecasting results are the average MSE results on lengths {96, 192, 336, 720}.

实验结果

研究问题

• RQ1 基于卷积的架构结合自适应谱处理，是否能够在时间序列分类、预测和异常检测方面达到或超过基于 Transformer 的模型？
• RQ2 自适应频域滤波如何影响噪声鲁棒性以及长程/短程依赖建模？
• RQ3 交互卷积块在捕捉多尺度时序模式方面的影响是什么？
• RQ4 自监督预训练是否在多样化时间序列数据集上提升性能和鲁棒性？

主要发现

• TSLANet 在分类、预测和异常检测任务中实现了优于若干最新基线的性能。
• 具有自适应高频阈值的自适应谱块（ASB）在降低噪声的同时保留了相关的谱信息。
• 交互卷积块通过使多尺度卷积之间发生交互，提升了特征提取。
• 自监督预训练进一步提升了跨数据集的表示质量和鲁棒性。
• TSLANet 在 FLOPs 和参数量方面相比某些 Transformer 基模型具有竞争力的准确性。
• 消融研究表明在去除 ASB 或 ICB 时性能显著下降，验证了它们的贡献。

Figure 2 : The structure of our proposed TSLANet . The input time series is split into patches, and positional embeddings are added. Next, the output embeddings pass through TSLANet layers, where each layer consists of two main components. The first is the Adaptive Spectral Block, which leverages fr