[Paper Review] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

Tian Zhou, Ziqing Ma+4 more|arXiv (Cornell University)|Jan 30, 2022

Time Series Analysis and Forecasting535 citations

TL;DR

FEDformer integrates seasonal-trend decomposition with frequency-domain Transformer blocks to achieve linear-complexity long-term forecasting and improved accuracy over state-of-the-art methods.

ABSTRACT

Although Transformer-based methods have significantly improved state-of-the-art results for long-term series forecasting, they are not only computationally expensive but more importantly, are unable to capture the global view of time series (e.g. overall trend). To address these problems, we propose to combine Transformer with the seasonal-trend decomposition method, in which the decomposition method captures the global profile of time series while Transformers capture more detailed structures. To further enhance the performance of Transformer for long-term prediction, we exploit the fact that most time series tend to have a sparse representation in well-known basis such as Fourier transform, and develop a frequency enhanced Transformer. Besides being more effective, the proposed method, termed as Frequency Enhanced Decomposed Transformer ({\bf FEDformer}), is more efficient than standard Transformer with a linear complexity to the sequence length. Our empirical studies with six benchmark datasets show that compared with state-of-the-art methods, FEDformer can reduce prediction error by $14.8\%$ and $22.6\%$ for multivariate and univariate time series, respectively. Code is publicly available at https://github.com/MAZiqing/FEDformer.

Motivation & Objective

• Motivate long-term forecasting with Transformer-based models to capture global time-series properties.
• Propose a frequency-domain Transformer architecture that combines Fourier/Wavelet representations with seasonal-trend decomposition.
• Achieve linear time/memory complexity via random subset frequency components and mixture-of-experts decomposition.
• Demonstrate improvements over state-of-the-art methods on six real-world datasets across multiple domains.

Proposed method

• Introduce Frequency Enhanced Block (FEB) and Frequency Enhanced Attention (FEA) to operate in the Fourier or Wavelet domain as substitutes for self-attention and cross-attention.
• Incorporate a Mixture Of Experts Decomposition (MOEDecomp) to extract and combine seasonal-trend components.
• Use random selection of a fixed number of Fourier components (or wavelet equivalents) to obtain a compact time-series representation with theoretical guarantees (Theorem 1).
• Provide two variants FEB-f/FEA-f (Fourier) and FEB-w/FEA-w (Wavelet) to learn in frequency domains.
• Combine decoded seasonal components with trend components to generate final forecasts.
• Analyze computational complexity showing O(L) time and memory with fixed mode counts, and O(N log N) reductions relative to full attention where applicable.

Experimental results

Research questions

• RQ1 Can decomposing time series into seasonal/trend components improve long-term forecasting when combined with Transformer architectures?
• RQ2 Does operating in the frequency domain (Fourier or Wavelet) enable efficient, scalable long-horizon forecasting without sacrificing accuracy?
• RQ3 Can a random, sparse selection of Fourier/Wavelet components preserve essential history for accurate predictions (Theorem 1)?:
• RQ4 What is the impact of integrating a mixture-of-experts decomposition on extracting seasonal-trend patterns?
• RQ5 How does FEDformer perform against state-of-the-art Transformer-based models on diverse real-world datasets?

Key findings

• FEDformer achieves lower prediction error than state-of-the-art methods by 14.8% (multivariate) and 22.6% (univariate) on six benchmarks.
• FEB/FEA blocks can replace self-attention and cross-attention with improved or maintained performance across experiments.
• MOEDecomp provides additional gains (~2.96% average) over single-decomposition schemes.
• FEDformer shows strong gains across horizons (96, 192, 336, 720 steps) and datasets (energy, traffic, economics, weather, disease).
• Two frequency variants (Fourier and Wavelet) offer complementary strengths, suggesting using both for robust long-term forecasting.