[Paper Review] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

Yong Liu, Tengge Hu+5 more|arXiv (Cornell University)|Oct 10, 2023

Time Series Analysis and Forecasting354 citations

TL;DR

iTransformer inverts transformer architecture by treating each variate as an independent token and applying self-attention across variates while using a shared FFN for series representations, achieving state-of-the-art time series forecasting.

ABSTRACT

The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

Motivation & Objective

• Question the necessity of traditional Transformer embeddings for multivariate time series.
• Propose an inverted-Transformer design that embeds each variate as its own token and applies attention across variates.
• Show that inverted architecture improves performance, generalization across variates, and effective use of longer lookback windows.
• Demonstrate state-of-the-art results on real-world forecasting benchmarks and analyze component choices.

Proposed method

• Embed each variate as an independent token (variates as tokens).
• Use self-attention to capture multivariate correlations across variate tokens.
• Apply a shared feed-forward network to each variate token to learn series representations.
• Use layer normalization applied to per-variate representations to reduce measurement discrepancies.
• Predict future values via a simple projection from the final per-variate representation.

Experimental results

Research questions

• RQ1 Can inverting the Transformer architecture (treating each variate as a separate token) improve multivariate time series forecasting?
• RQ2 Do inverted Transformer components (attention across variates and FFN per variate) lead to better representations and predictive performance?
• RQ3 How does iTransformer generalize to unseen variates and handle varying lookback windows?
• RQ4 What is the impact of lookback length on performance for inverted vs vanilla Transformer forecasters?
• RQ5 Is iTransformer a viable backbone for real-world, high-dimensional time series forecasting?

Key findings

• iTransformer achieves state-of-the-art performance across several real-world datasets.
• Inversion enables attention to learn multivariate correlations more clearly and FFNs to learn variate-specific representations.
• The model generalizes better to unseen variates and supports flexible numbers of variates between training and inference.
• Longer lookback windows improve performance for iTransformer, whereas vanilla Transformer benefits are limited or negative.
• Applying efficient attention variants within the inverted framework yields strong performance with reduced computation.
• Ablation studies show the combination of variate-level attention and temporal FFN yields the best results.