[Paper Review] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

Haixu Wu, Jiehui Xu+2 more|arXiv (Cornell University)|Jun 24, 2021

Neural Networks and ApplicationsComputer Science1,311 citations

TL;DR

Autoformer introduces a decomposition-based transformer architecture with an Auto-Correlation mechanism to achieve efficient, accurate long-term time series forecasting, with complexity O(L log L) and state-of-the-art results on six benchmarks.

ABSTRACT

Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: \\url{https://github.com/thuml/Autoformer}.

Motivation & Objective

• Address the challenges of long-term time series forecasting by modeling intricate temporal patterns.
• Improve efficiency of Transformer-based forecasting for long sequences without sacrificing information usage.
• Introduce a progressive decomposition framework that operates inside the model rather than as pre-processing.
• Leverage series periodicity to design an Auto-Correlation mechanism that aggregates sub-series at the sub-series level.

Proposed method

• Propose Autoformer, a decomposition forecasting architecture that embeds a SeriesDecomp inner block to separate trend-cyclical and seasonal components.
• Replace standard self-attention with an Auto-Correlation mechanism that uses series autocorrelation to identify period-based dependencies.
• Compute autocorrelation via FFT (Wiener–Khinchin theorem) to achieve O(L log L) complexity and roll sub-series by selected time delays for aggregation.
• Structure the encoder to model seasonal components and progressively refine trend information in the decoder through accumulation of trend parts.
• Incorporate an inner decomposition block within each layer to progressively decompose hidden variables during forecasting.

Experimental results

Research questions

• RQ1 How can decomposition and periodicity be integrated into deep forecasting models to handle long-term horizons?
• RQ2 Can a series-wise attention-like mechanism (Auto-Correlation) outperform point-wise self-attention in terms of accuracy and efficiency for long sequences?
• RQ3 Does progressive decomposition within a deep model improve long-term forecast robustness and accuracy across multiple real-world domains?
• RQ4 What are the empirical gains of Autoformer on standard long-term forecasting benchmarks compared to state-of-the-art Transformer-based methods?

Key findings

• Autoformer achieves state-of-the-art accuracy on six benchmarks spanning energy, traffic, economics, weather and disease.
• The proposed Auto-Correlation mechanism provides O(L log L) complexity and superior information utilization by aggregating sub-series at the period level.
• Autoformer demonstrates strong long-term robustness, with performance that degrades more gracefully as prediction length increases.
• A progressive decomposition architecture improves performance when integrated with other models, indicating broad applicability of the decomposition approach.
• Across settings, Autoformer attains substantial MSE reductions (e.g., up to 38% averaged) relative to baselines on multiple datasets.