[论文解读] Transformers in Time Series: A Survey

Qingsong Wen, Tian Zhou+5 more|arXiv (Cornell University)|Feb 15, 2022

Time Series Analysis and Forecasting被引用 61

一句话总结

A comprehensive survey of Transformer-based methods for time series, detailing network adaptations, applications (forecasting, anomaly detection, classification), empirical analyses, and future directions.

ABSTRACT

Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance. To the best of our knowledge, this paper is the first work to comprehensively and systematically summarize the recent advances of Transformers for modeling time series data. We hope this survey will ignite further research interests in time series Transformers.

研究动机与目标

• 激励使用 Transformers 来捕捉时间序列数据中的长程依赖。
• 总结用于时间序列建模的网络级适配和架构方面的变化。
• 按预测、异常检测和分类任务对基于 Transformer 的方法进行分类。
• 提供关于鲁棒性、模型规模以及季节-趋势分解效应的实证见解。
• 突出时间序列 Transformers 的开放挑战与未来方向。

提出的方法

• 介绍基础的 Transformer 初步概念及位置编码策略。
• 讨论注意力模块、效率策略以及面向时间序列的架构层创新。
• 给出基于网络修改和应用的分类法。
• 调查用于预测、异常检测和分类的模块级和架构级变体。
• 就输入长度、模型规模和季节-趋势分解的鲁棒性进行实证分析。
• 总结实用指南与未来研究方向。

实验结果

研究问题

• RQ1 哪些主要的 Transformer 适配能够有效建模时间序列的依赖性和季节性？
• RQ2 基于 Transformer 的时间序列模型在预测、异常检测和分类任务中的表现如何，以及它们的鲁棒性/模型规模权衡？
• RQ3 哪些未来方向可以进一步提升时间序列 Transformer 的性能（归纳偏置、GNN 集成、预训练、架构变体）？

主要发现

• 基于 Transformer 的时间序列模型在预测、异常检测和分类方面提供了有利的长程依赖建模。
• 面向效率的注意力（稀疏/低秩）以及多分辨率或分层体系结构提升了对长序列的处理能力。
• 季节-趋势分解可以显著提升预测性能（在报道的实验中达到 50%-80%）。
• 模型规模和深度并不总是带来更好的预测结果；中等规模的配置有时优于非常深的模型。
• 时间序列的预训练研究较少，显示出潜力但仍处于初期阶段。
• 将 Transformers 与 GNNs 相结合并引入专门针对时间序列的归纳偏置可以提升性能。