[Paper Review] One Fits All:Power General Time Series Analysis by Pretrained LM

Tian Zhou, PeiSong Niu+3 more|arXiv (Cornell University)|Feb 23, 2023

Time Series Analysis and ForecastingComputer Science117 citations

TL;DR

This paper demonstrates that a frozen pretrained transformer from NLP/CV can achieve state-of-the-art or competitive performance across diverse time series tasks by fine-tuning only lightweight components. It also analyzes self-attention behavior and links it to PCA to explain universality across domains.

ABSTRACT

Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer.The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.

Motivation & Objective

• Motivate and enable general time series analysis with a foundation model trained on large-scale language or vision data.
• Propose a unified framework that adapts a frozen pretrained transformer to time series tasks without modifying core attention/FFN blocks.
• Show that cross-domain pretrained models achieve competitive to state-of-the-art results across multiple time series tasks.
• Provide theoretical and empirical insight into why self-attention behaves similarly to PCA and enables universality across domains.

Proposed method

• Use a frozen pretrained transformer (GPT-2 backbone) for time series tasks, freezing self-attention and FFN blocks while training only embedding, normalization, and output layers.
• Design an input embedding layer to project time series into the pretrained model via linear probing.
• Apply data normalization and a patching mechanism to form patch-based tokens for local semantic information.
• Fine-tune the model on diverse time series tasks (classification, short/long-term forecasting, imputation, anomaly detection, few-shot/zero-shot forecasting).
• Experiment with multiple pretrained backbones (GPT-2, BERT, BEiT) to demonstrate cross-domain universality.

Experimental results

Research questions

• RQ1 Can a frozen pretrained language/vision transformer achieve competitive performance on general time series analysis tasks without altering core transformer blocks?
• RQ2 To what extent does cross-domain pretrained knowledge transfer enable time series forecasting, classification, anomaly detection, and imputation?
• RQ3 Why might self-attention in pretrained transformers emulate PCA-like representations, supporting universal compute across domains?
• RQ4 Do cross-modality pretrained models (language, vision) provide universal effectiveness for time series tasks, including zero-shot and few-shot settings?

Key findings

• GPT2-backbone FPT achieves on-par or state-of-the-art performance across major time series tasks, including forecasting, classification, anomaly detection, and imputation.
• Imputation experiments show GPT2(3) FPT often yields the best results, reducing MSE by notable margins on several datasets.
• Classification and anomaly detection results show GPT2(6) FPT outperforming or matching baselines on multiple benchmarks and datasets.
• Few-shot and zero-shot forecasting demonstrate that GPT2(6) FPT maintains strong performance without extensive task-specific data.
• Analyses indicate self-attention behaves similarly to PCA, providing a theoretical link between transformer attention and principal components, supporting the universality claim.
• Experiments with BERT and BEiT backbones suggest the universality of cross-domain pretrained models for time series tasks.