[Paper Review] Timer: Generative Pre-trained Transformers Are Large Time Series Models

Yong Liu, Haoran Zhang+4 more|arXiv (Cornell University)|Feb 4, 2024

Time Series Analysis and Forecasting4 citations

TL;DR

This paper introduces Timer, a large-scale, decoder-only Transformer model pre-trained on 1 billion time points using a unified single-series sequence (S3) format, enabling few-shot generalization across time series forecasting, imputation, and anomaly detection. By adopting a GPT-style autoregressive objective, Timer achieves state-of-the-art performance with minimal fine-tuning, demonstrating strong scalability and task generality in low-data regimes.

ABSTRACT

Deep learning has contributed remarkably to the advancement of time series analysis. Still, deep models can encounter performance bottlenecks in real-world data-scarce scenarios, which can be concealed due to the performance saturation with small models on current benchmarks. Meanwhile, large models have demonstrated great powers in these scenarios through large-scale pre-training. Continuous progress has been achieved with the emergence of large language models, exhibiting unprecedented abilities such as few-shot generalization, scalability, and task generality, which are however absent in small deep models. To change the status quo of training scenario-specific small models from scratch, this paper aims at the early development of large time series models (LTSM). During pre-training, we curate large-scale datasets with up to 1 billion time points, unify heterogeneous time series into single-series sequence (S3) format, and develop the GPT-style architecture toward LTSMs. To meet diverse application needs, we convert forecasting, imputation, and anomaly detection of time series into a unified generative task. The outcome of this study is a Time Series Transformer (Timer), which is generative pre-trained by next token prediction and adapted to various downstream tasks with promising capabilities as an LTSM. Code and datasets are available at: https://github.com/thuml/Large-Time-Series-Model.

Motivation & Objective

• To address the performance bottleneck of small models in data-scarce time series scenarios by developing large time series models (LTSMs) with strong few-shot generalization.
• To overcome the lack of scalable, unified architectures and pre-training strategies for time series, mirroring the success of large language models.
• To unify diverse time series tasks—forecasting, imputation, anomaly detection—into a single generative modeling framework using a common model and training objective.
• To establish a large-scale, multi-domain dataset (UTSD) with 1 billion time points to enable robust pre-training and scalability studies.
• To demonstrate that decoder-only Transformers trained via autoregressive next-token prediction can achieve superior scalability and task generality in time series analysis.

Proposed method

• Curated a Unified Time Series Dataset (UTSD) with 1 billion time points from diverse domains, enabling scalable pre-training.
• Proposed the single-series sequence (S3) format to unify heterogeneous time series into a standardized token sequence format.
• Designed a decoder-only Transformer architecture trained via autoregressive next-token prediction, emulating GPT-style pre-training.
• Unified forecasting, imputation, and anomaly detection as a single generative task, enabling zero-shot and few-shot adaptation.
• Applied data-centric pre-training with masked and contrastive objectives during pre-training, followed by fine-tuning on downstream tasks.
• Scaled model size and pre-training data size to validate the scaling laws of the model, showing consistent performance gains.

Experimental results

Research questions

• RQ1 Can large-scale pre-training of time series models via autoregressive generation achieve strong few-shot generalization in low-data regimes?
• RQ2 Does a decoder-only Transformer architecture trained in a GPT-style manner outperform encoder-only models in time series tasks?
• RQ3 Can a unified generative framework effectively handle diverse time series tasks such as forecasting, imputation, and anomaly detection with a single model?
• RQ4 How does model and data scale affect performance in time series modeling, and does the scaling law hold for large time series models?
• RQ5 Can a standardized S3 format unify heterogeneous time series data for scalable, large-scale pre-training?

Key findings

• Timer achieves state-of-the-art performance on forecasting, imputation, and anomaly detection tasks with only 5% of training data, outperforming existing models.
• On the PEMS03 dataset with 5% training samples, Timer reduced MSE from 0.188 (baseline) to 0.125 with 12G pre-training, showing strong few-shot generalization.
• With 20% training data, Timer achieved a 25% relative improvement in MSE on PEMS08 compared to baseline models, demonstrating scalability.
• The decoder-only architecture with GPT-style pre-training showed superior performance over encoder-only models, especially in low-data regimes.
• Scaling model size from 4G to 12G parameters reduced MSE on PEMS03 by 30% at 5% data, confirming the validity of scaling laws in time series.
• Visualizations confirmed that Timer accurately generates missing values in imputation, predicts future sequences in forecasting, and detects anomalies by generating out-of-distribution outputs.