[論文レビュー] Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Shiyang Li, Xiaoyong Jin+5 more|arXiv (Cornell University)|Jun 29, 2019

Time Series Analysis and Forecasting参考文献 35被引用数 1,006

ひとこと要約

本論文は、convolutional self-attention と LogSparse Transformer を導入することで局所的な文脈認識を高め、メモリコストを削減し、メモリ制約下で長期依存性を持つ Transformer ベースの時系列予測を実現します。

ABSTRACT

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

研究の動機と目的

• Motivate the use of Transformer architectures for time series forecasting to capture long- and short-term dependencies.
• Address locality-agnostic self-attention by incorporating local context via causal convolution.
• Mitigate memory bottlenecks of standard Transformers to enable modeling long, fine-grained time series.
• Demonstrate improved forecasting performance on synthetic and real-world datasets under constrained memory.

提案手法

• Introduce convolutional self-attention by generating queries and keys through causal convolution to incorporate local context.
• Generalize canonical self-attention with kernel k, where k=1 recovers standard attention.
• Propose LogSparse Transformer with attention limited to O(log L) previous positions per cell, yielding O(L (log L)^2) memory.
• Theoretically show that with O(log L) layers information can flow from any past position to any current position.
• Expose local attention and restart attention variants to further improve information flow and efficiency.
• Empirically compare against baselines on synthetic and real datasets, including rolling-window forecasts and horizon-based tasks.

実験結果

リサーチクエスチョン

• RQ1 Can convolutional self-attention improve locality awareness and forecasting accuracy over standard Transformer in time series?
• RQ2 Does LogSparse Transformer substantially reduce memory usage while preserving or improving predictive performance for long, fine-grained time series?
• RQ3 How do kernel size and sparsity patterns affect learning dynamics and forecasting accuracy across datasets with varying long-term dependencies?
• RQ4 What is the impact of locality-aware attention on training convergence and model efficiency compared to full attention?

主な発見

• Convolutional self-attention improves forecasting accuracy by leveraging local context in query-key matching.
• LogSparse Transformer achieves O(L (log L)^2) memory, enabling long, fine-grained time series modeling under memory constraints.
• Larger kernel sizes in convolutional self-attention yield notable gains on challenging datasets with strong long-term dependencies.
• Experiments show favorable performance of the proposed methods against state-of-the-art baselines across synthetic and real-world datasets.
• Convolutional self-attention accelerates training and reduces training loss, suggesting easier optimization.