[Paper Review] Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting

Kashif Rasul, Arjun Ashok+16 more|arXiv (Cornell University)|Oct 12, 2023

Stock Market Forecasting Methods32 citations

TL;DR

Lag-Llama is a decoder-only transformer foundation model for univariate probabilistic time series forecasting pretrained on a large, diverse corpus; it shows strong zero-shot generalization and state-of-the-art few-shot fine-tuning performance across unseen datasets.

ABSTRACT

Over the past years, foundation models have caused a paradigm shift in machine learning due to their unprecedented capabilities for zero-shot and few-shot generalization. However, despite the success of foundation models in modalities such as natural language processing and computer vision, the development of foundation models for time series forecasting has lagged behind. We present Lag-Llama, a general-purpose foundation model for univariate probabilistic time series forecasting based on a decoder-only transformer architecture that uses lags as covariates. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities compared to a wide range of forecasting models on downstream datasets across domains. Moreover, when fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance, outperforming prior deep learning approaches, emerging as the best general-purpose model on average. Lag-Llama serves as a strong contender to the current state-of-art in time series forecasting and paves the way for future advancements in foundation models tailored to time series data.

Motivation & Objective

• Motivate the use of foundation models for probabilistic time series forecasting to enable zero-shot and few-shot transfer across diverse domains.
• Develop a simple, scalable architecture that uses lag covariates within a decoder-only transformer.
• Pretrain on a large, diverse corpus of univariate time series to study generalization across frequencies and domains.
• Evaluate zero-shot, fine-tuning, and few-shot adaptation on unseen downstream datasets against strong baselines.

Proposed method

• Use a decoder-only transformer (inspired by LLaMA) with lag-based tokenization plus covariates.
• Construct lag features from a prescribed set of lag indices plus date-time covariates.
• Train to output parameters of a probabilistic distribution (Student's t) via a distribution head.
• Apply robust standardization (median and IQR) with per-series summary covariates (mean, std) for scaling.
• Pretrain on 7,965 time series (~352 million windows) across six domains; fine-tune on unseen datasets with early stopping based on validation loss.
• Evaluate via autoregressive sampling to generate predictive distributions and compute CRPS; report zero-shot and few-shot performance.

Experimental results

Research questions

• RQ1 How well can a foundation model pretrained on diverse time series generalize to unseen datasets without any downstream fine-tuning (zero-shot)?
• RQ2 What performance does Lag-Llama achieve after fine-tuning on unseen datasets, and how does it compare to dataset-specific and other general-purpose models?
• RQ3 Can Lag-Llama adapt effectively in few-shot regimes when limited historical data is available from new domains?
• RQ4 How does the diversity and size of the pretraining corpus influence Lag-Llama’s forecasting performance and scaling behavior?

Key findings

• Lag-Llama achieves competitive zero-shot performance on unseen datasets (average CRPS ranking 6.714 across datasets).
• Fine-tuned Lag-Llama attains state-of-the-art performance on several datasets and achieves the best average rank (2.786) among compared methods.
• Lag-Llama outperforms many baselines (including Informer, AutoFormer, ETSFormer) as a strong general-purpose model when pretrained from scratch on diverse data.
• In few-shot settings with varying available history (20–80%), Lag-Llama consistently provides strong adaptation, often outperforming baselines, with performance improving as history increases.
• Lag-Llama can handle unseen frequencies and domains, with comparable zero-shot performance on some domains (e.g., exchange-rate) and close-to-SOTA performance after fine-tuning.