[Paper Review] ETSformer: Exponential Smoothing Transformers for Time-series Forecasting

Gerald Woo, Chenghao Liu+3 more|arXiv (Cornell University)|Feb 3, 2022

Time Series Analysis and Forecasting108 citations

TL;DR

ETSformer introduces Exponential Smoothing Attention and Frequency Attention to Transformers for time-series forecasting, enabling interpretable level/growth/seasonal decomposition and achieving state-of-the-art results on multiple real-world datasets.

ABSTRACT

Transformers have been actively studied for time-series forecasting in recent years. While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations, e.g., they generally lack of decomposition capability and interpretability, and are neither effective nor efficient for long-term forecasting. In this paper, we propose ETSFormer, a novel time-series Transformer architecture, which exploits the principle of exponential smoothing in improving Transformers for time-series forecasting. In particular, inspired by the classical exponential smoothing methods in time-series forecasting, we propose the novel exponential smoothing attention (ESA) and frequency attention (FA) to replace the self-attention mechanism in vanilla Transformers, thus improving both accuracy and efficiency. Based on these, we redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time-series components such as level, growth and seasonality. Extensive experiments on various time-series benchmarks validate the efficacy and advantages of the proposed method. Code is available at https://github.com/salesforce/ETSformer.

Motivation & Objective

• Incorporate inductive biases of time-series structure (level, growth, seasonality) into a Transformer for forecasting.
• Replace vanilla self-attention with Exponential Smoothing Attention (ESA) and Frequency Attention (FA) to capture recentness and seasonality.
• Develop a modular encoder-decoder architecture that learns interpretable latent components and outputs a forecast as a sum of level, growth, and seasonal terms.
• Demonstrate scalability and efficiency with O(L log L) complexity and validate on diverse real-world datasets.

Proposed method

• Propose Exponential Smoothing Attention (ESA) whose weights follow an exponential decay based on relative time lag, enabling O(L log L) computation via a fast cross-correlation/FFT-based algorithm.
• Propose Frequency Attention (FA) using Fourier bases to identifiy and extrapolate dominant seasonal patterns with O(L log L) complexity.
• Build a Multi-Head ESA (MH-ESA) to extract latent growth signals from residuals of the lookback window.
• Decompose each encoder layer into Seasonal (FA), Growth (MH-ESA), and Level components with learnable smoothing parameters, then progressively refine residuals.
• Decode forecasts as a composition of Level, Growth, and Seasonal terms across N layers, including growth damping for robust multi-step forecasts.

Experimental results

Research questions

• RQ1 Can time-series specific inductive biases (level, growth, seasonality) be integrated into a Transformer to improve forecasting accuracy and interpretability?
• RQ2 Do Exponential Smoothing Attention and Frequency Attention provide efficiency and performance benefits over vanilla attention in time-series forecasting?
• RQ3 Can a modular, decomposed architecture learn meaningful level, growth, and seasonal representations that align with classical exponential smoothing?
• RQ4 Is the proposed ETSformer scalable to long lookback windows while achieving competitive or superior accuracy on multivariate and univariate datasets?

Key findings

• ETSformer achieves state-of-the-art performance, outperforming baselines on 35 out of 40 multivariate settings and 17 out of 23 univariate settings.
• The model remains highly competitive even when not best, ranking in the top-2 across all 40 multivariate settings and 21 of 23 univariate settings.
• Ablation shows removing any of Level, Growth, or Season components degrades performance, with Level being particularly critical.
• ETSformer provides interpretable forecasts by decomposing predictions into level, growth, and seasonal components, which align with ground-truth patterns on synthetic data and real datasets.
• The ESA and FA mechanisms deliver O(L log L) complexity, enabling efficient long-sequence inference.