[Paper Review] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Yuqi Nie, Nam Hoai Nguyen+2 more|arXiv (Cornell University)|Nov 27, 2022

Time Series Analysis and ForecastingComputer Science534 citations

TL;DR

PatchTST introduces time-series patching and channel-independence in Transformer models to improve long-term forecasting and self-supervised representation learning for multivariate time series.

ABSTRACT

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

Motivation & Objective

• Motivate improving long-term forecasting accuracy for multivariate time series with Transformer architectures.
• Explore efficiency and memory benefits by reducing input tokens through patching.
• Investigate channel-independence as a weight-sharing strategy across series channels.
• Demonstrate self-supervised representation learning with patch-level masking and transfer learning capabilities.

Proposed method

• Divide multivariate time series into per-channel univariate sequences that share embedding and Transformer weights (channel-independence).
• Segment each univariate sequence into patches to form input tokens, reducing attention complexity from O(N^2) to O((L/S)^2)-like behavior.
• Use a vanilla Transformer encoder with patch-based embeddings and position encodings to forecast future values.
• Apply instance normalization per time series before patching to stabilize distributions.
• Train with MSE loss across channels, with a fusion head to produce multi-step forecasts.
• For self-supervised learning, mask a subset of patches and train the model to reconstruct them (no forecasting head).
• Evaluate both supervised forecasting and self-supervised pre-training with transfer learning scenarios.

Experimental results

Research questions

• RQ1 Does patch-based tokenization improve information retention and forecasting accuracy compared to point-wise tokens in time-series transformers?
• RQ2 Does enforcing channel-independence (per-channel tokens with shared weights) improve scalability and performance for multivariate series?
• RQ3 Can self-supervised PatchTST representations transfer across datasets and outperform supervised baselines or other Transformer-based models?
• RQ4 How do longer look-back windows interact with patching to affect forecasting accuracy and efficiency?

Key findings

• PatchTST achieves significant accuracy gains over state-of-the-art Transformer baselines on multiple datasets (e.g., up to 21.0% MSE reduction and 16.7% MAE reduction for PatchTST/64 over best Transformer baselines).
• Patching reduces input tokens and attention memory/computation, enabling longer look-back windows to improve forecasting (e.g., L increased from 96 to 336 improves MSE from 0.518 to 0.397).
• Channel-independence coupled with patching contributes to improved performance and scalable architecture across multiple datasets.
• Self-supervised PatchTST yields state-of-the-art representation learning and transfer learning performance, often surpassing supervised training on large datasets.
• Pre-training on one dataset and fine-tuning or transferring representations to others can maintain or improve forecasting accuracy compared to training from scratch.