[Paper Review] Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Shiyang Li, Xiaoyong Jin+5 more|arXiv (Cornell University)|Jun 29, 2019

Time Series Analysis and Forecasting35 references1,006 citations

TL;DR

The paper introduces convolutional self-attention and LogSparse Transformer to enhance local context awareness and reduce memory costs, enabling Transformer-based time series forecasting with long-term dependencies under memory constraints.

ABSTRACT

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

Motivation & Objective

• Motivate the use of Transformer architectures for time series forecasting to capture long- and short-term dependencies.
• Address locality-agnostic self-attention by incorporating local context via causal convolution.
• Mitigate memory bottlenecks of standard Transformers to enable modeling long, fine-grained time series.
• Demonstrate improved forecasting performance on synthetic and real-world datasets under constrained memory.

Proposed method

• Introduce convolutional self-attention by generating queries and keys through causal convolution to incorporate local context.
• Generalize canonical self-attention with kernel k, where k=1 recovers standard attention.
• Propose LogSparse Transformer with attention limited to O(log L) previous positions per cell, yielding O(L (log L)^2) memory.
• Theoretically show that with O(log L) layers information can flow from any past position to any current position.
• Expose local attention and restart attention variants to further improve information flow and efficiency.
• Empirically compare against baselines on synthetic and real datasets, including rolling-window forecasts and horizon-based tasks.

Experimental results

Research questions

• RQ1 Can convolutional self-attention improve locality awareness and forecasting accuracy over standard Transformer in time series?
• RQ2 Does LogSparse Transformer substantially reduce memory usage while preserving or improving predictive performance for long, fine-grained time series?
• RQ3 How do kernel size and sparsity patterns affect learning dynamics and forecasting accuracy across datasets with varying long-term dependencies?
• RQ4 What is the impact of locality-aware attention on training convergence and model efficiency compared to full attention?

Key findings

• Convolutional self-attention improves forecasting accuracy by leveraging local context in query-key matching.
• LogSparse Transformer achieves O(L (log L)^2) memory, enabling long, fine-grained time series modeling under memory constraints.
• Larger kernel sizes in convolutional self-attention yield notable gains on challenging datasets with strong long-term dependencies.
• Experiments show favorable performance of the proposed methods against state-of-the-art baselines across synthetic and real-world datasets.
• Convolutional self-attention accelerates training and reduces training loss, suggesting easier optimization.