[Paper Review] Large Language Models Are Zero-Shot Time Series Forecasters

Nate Gruver, Marc Finzi+2 more|arXiv (Cornell University)|Oct 11, 2023

Topic Modeling94 citations

TL;DR

The paper demonstrates that large language models can forecast time series in a zero-shot setting by encoding numbers as text and treating forecasting as next-token text generation, achieving competitive or superior results to purpose-built models without fine-tuning.

ABSTRACT

By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large language models (LLMs) such as GPT-3 and LLaMA-2 can surprisingly zero-shot extrapolate time series at a level comparable to or exceeding the performance of purpose-built time series models trained on the downstream tasks. To facilitate this performance, we propose procedures for effectively tokenizing time series data and converting discrete distributions over tokens into highly flexible densities over continuous values. We argue the success of LLMs for time series stems from their ability to naturally represent multimodal distributions, in conjunction with biases for simplicity, and repetition, which align with the salient features in many time series, such as repeated seasonal trends. We also show how LLMs can naturally handle missing data without imputation through non-numerical text, accommodate textual side information, and answer questions to help explain predictions. While we find that increasing model size generally improves performance on time series, we show GPT-4 can perform worse than GPT-3 because of how it tokenizes numbers, and poor uncertainty calibration, which is likely the result of alignment interventions such as RLHF.

Motivation & Objective

• Demonstrate that LLMs can zero-shot forecast time series data without fine-tuning.
• Develop effective tokenization and continuous-density mapping for numeric sequences.
• Show that LLMs capture multimodal distributions and uncertainty in time series.
• Explore handling of missing data, side information, and explanation capabilities via prompting.
• Analyze how model scale and alignment affect forecasting performance and uncertainty calibration.

Proposed method

• Encode time series as strings of numerical digits for next-token prediction in LLMs.
• Design numeric tokenization strategies to ensure stable arithmetic and pattern learning across models (e.g., digit-level tokenization, spaces, and comma separators).
• Convert discrete token distributions into continuous densities via a hierarchical mixture of uniform bins to enable probabilistic forecasting and likelihood estimates.
• Approximate continuous likelihoods by placing uniform distributions in each digit-based bin and apply a change-of-variables for rescaling to original space.
• Sample multiple futures from the LLM and aggregate to form point forecasts and probabilistic forecasts (quantiles, CRPS, NLL).
• Compare zero-shot LLM forecasts against standard time series baselines on multiple datasets and analyze calibration across models.

Experimental results

Research questions

• RQ1 Can pretrained LLMs perform time series forecasting without fine-tuning when numbers are encoded as text?
• RQ2 How should time series data be tokenized to maximize forecasting accuracy and stable numerical reasoning in LLMs?
• RQ3 Can LLMs provide well-calibrated probabilistic forecasts and uncertainty estimates for time series?
• RQ4 What is the impact of model scale and alignment (e.g., RLHF) on forecasting performance and uncertainty calibration?
• RQ5 How do missing data, textual side information, and explanation prompts affect forecasting with LLMs?

Key findings

• LLMTime can exceed or match purpose-built time series methods in zero-shot forecasting across diverse datasets.
• Proper numeric tokenization and rescaling are crucial for good performance and numerical reasoning in LLMs.
• LLMs can represent multimodal and uncertain continuations through digit-based continuous densities, improving probabilistic forecasts.
• GPT-3 and LLaMA-2 70B show strong deterministic performance, with probabilistic metrics like NLL and CRPS often outperforming baselines.
• GPT-4 and chat-based LLMs may exhibit worse calibration or forecasting performance due to alignment interventions (RLHF) and tokenization quirks.
• Uncertainty representation improves with base model quality, while zero-shot performance scales with model size.