[Paper Review] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Haoyi Zhou, Shanghang Zhang+5 more|arXiv (Cornell University)|Dec 14, 2020

Time Series Analysis and Forecasting57 references462 citations

TL;DR

Informer introduces ProbSparse self-attention, attention distilling, and a generative style decoder to enable efficient and scalable long sequence time-series forecasting with Transformer-like models.

ABSTRACT

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism, which achieves $O(L \log L)$ in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

Motivation & Objective

• Motivate long sequence time-series forecasting (LSTF) and its prediction capacity challenges.
• Develop a Transformer-based model that is computationally and memory-efficient for LSTF.
• Propose mechanisms to improve long-range dependency capture without quadratic cost.
• Demonstrate practical, scalable forecasting on large real-world datasets.

Proposed method

• Replace canonical self-attention with ProbSparse self-attention that achieves O(L log L) time and memory.
• Introduce self-attention distilling to highlight dominating attention and reduce memory via layer-wise down-sampling.
• Use a generative style decoder to predict long output sequences in a single forward pass, reducing inference time and error accumulation.
• Provide an encoder-decoder architecture tailored for LSTF with input representations that enhance global and local temporal context.
• Train with MSE loss across target sequences and evaluate on univariate and multivariate forecasting tasks.

Experimental results

Research questions

• RQ1 Can Transformer-like models be made computation- and memory-efficient for very long input/output sequences in time-series forecasting?
• RQ2 Do ProbSparse self-attention, attention distilling, and a generative decoder jointly improve accuracy and efficiency for LSTF?
• RQ3 How does Informer perform on univariate vs. multivariate long horizon forecasting across real-world datasets?

Key findings

• Informer significantly improves forecasting performance across multiple datasets and horizon lengths.
• ProbSparse self-attention reduces computation and memory from quadratic to near-linear while maintaining competitive dependency alignment.
• Self-attention distilling sharply reduces encoder memory while preserving or improving long-range information processing.
• A generative style decoder enables long-sequence outputs with a single forward pass, accelerating inference and mitigating error propagation.
• Ablation studies show the effectiveness of the ProbSparse mechanism and the distilling approach under various configurations.