[Paper Review] Are Transformers Effective for Time Series Forecasting?

Ailing Zeng, Muxi Chen+2 more|arXiv (Cornell University)|May 26, 2022

Time Series Analysis and Forecasting182 citations

TL;DR

The paper shows that a simple one-layer linear baseline (LTSF-Linear) often outperforms state-of-the-art Transformer-based long-term time series forecasting models across nine real-world datasets, questioning the temporal modeling efficacy of Transformers for LTSF.

ABSTRACT

Recently, there has been a surge of Transformer-based solutions for the long-term time series forecasting (LTSF) task. Despite the growing performance over the past few years, we question the validity of this line of research in this work. Specifically, Transformers is arguably the most successful solution to extract the semantic correlations among the elements in a long sequence. However, in time series modeling, we are to extract the temporal relations in an ordered set of continuous points. While employing positional encoding and using tokens to embed sub-series in Transformers facilitate preserving some ordering information, the nature of the \emph{permutation-invariant} self-attention mechanism inevitably results in temporal information loss. To validate our claim, we introduce a set of embarrassingly simple one-layer linear models named LTSF-Linear for comparison. Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin. Moreover, we conduct comprehensive empirical studies to explore the impacts of various design elements of LTSF models on their temporal relation extraction capability. We hope this surprising finding opens up new research directions for the LTSF task. We also advocate revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future. Code is available at: \url{https://github.com/cure-lab/LTSF-Linear}.

Motivation & Objective

• Question the effectiveness of Transformer-based solutions for long-term time series forecasting (LTSF).
• Propose a simple linear baseline (LTSF-Linear) to challenge Transformer-based LTSF models.
• Evaluate and compare Transformer-based LTSF models and LTSF-Linear across diverse real-world datasets.
• Investigate design elements (embedding, look-back window, DMS vs IMS) and their impact on temporal relation extraction.

Proposed method

• Introduce LTSF-Linear, a one-layer temporal linear model that predicts future values directly via a weighted sum: X_hat = W X, with W in R^{T x L}.
• Develop two variants to handle trend and distribution shifts: DLinear (decomposition + linear predictors for trend and seasonal parts) and NLinear (normalization via subtracting the last value, then linear predictor).
• Benchmark against Transformer-based LTSF models (FEDformer, Autoformer, Informer, Pyraformer, LogTrans) on nine real-world multivariate datasets.
• Analyze the impact of look-back window size, input embeddings, and self-attention vs linear layers through ablations and input-shuffling experiments.
• Assess computational efficiency and practical inference costs.

Experimental results

Research questions

• RQ1 Do Transformer-based models truly capture temporal relations in long sequences for LTSF?
• RQ2 Are simple linear models competitive or superior for LTSF benchmarks, and under what conditions?
• RQ3 How do look-back window size and embedding choices affect temporal relation extraction in LTSF?
• RQ4 What is the role of DMS vs IMS forecasting in Transformer-based LTSF performance?
• RQ5 Do transformations like input shuffling reveal temporal order preservation differences between Transformer-based models and linear baselines?

Key findings

• LTSF-Linear outperforms existing Transformer-based LTSF models on nine benchmarks by about 20% to 50% in multivariate forecasting.
• Transformers often fail to reduce forecasting errors as look-back window size grows, suggesting limited temporal relation extraction for long sequences.
• LTSF-Linear remains strong even without modeling inter-variate correlations, indicating a powerful simple baseline for LTSF tasks.
• Ablation studies show self-attention and other Transformer components are not strictly necessary for these benchmarks, and simplifications can improve or match performance.
• Shuffling experiments reveal Transformer-based methods rely less on input order information than LTSF-Linear, implying weaker temporal ordering preservation in current Transformers.