[Paper Review] Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

Kashif Rasul, Calvin Seward+2 more|arXiv (Cornell University)|Jan 28, 2021

Time Series Analysis and Forecasting53 citations

TL;DR

TimeGrad is an autoregressive diffusion-based model for multivariate probabilistic time series forecasting that samples from the data distribution at each step and achieves state-of-the-art CRPS_sum on six real-world datasets.

ABSTRACT

In this work, we propose \ exttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experimentally that the proposed autoregressive denoising diffusion model is the new state-of-the-art multivariate probabilistic forecasting method on real-world data sets with thousands of correlated dimensions. We hope that this method is a useful tool for practitioners and lays the foundation for future research in this area.

Motivation & Objective

• Motivate multivariate probabilistic forecasting to capture dependencies across numerous time series dimensions.
• Develop an autoregressive energy-based model (EBM) using diffusion processes to model per-step distributions.
• Enable sampling from the predictive distribution via Langevin dynamics and autoregressive conditioning on past states.
• Evaluate TimeGrad against diverse baselines on large real-world datasets to demonstrate improved probabilistic forecasts.

Proposed method

• Use diffusion probabilistic models to learn the gradient of the log-density for per-time-step emissions.
• Condition the diffusion-based emission model on an autoregressive RNN (LSTM/GRU) state that encodes past observations and covariates.
• Train with a variational bound that reduces to a squared-error objective on predicted noise (ε_θ) similar to score-based models.
• Sample future steps via annealed Langevin dynamics to generate predictive trajectories.
• Normalize time series by context window means to stabilize training and inference.
• Incorporate covariate embeddings (categorical and continuous) to enrich conditioning information.

Experimental results

Research questions

• RQ1 Can TimeGrad model the full conditional distribution of future time steps in high-dimensional multivariate time series?
• RQ2 How does an autoregressive diffusion-based approach compare to VAR, VAR-Lasso, GARCH, KVAE, Vec-LSTM variants, and Transformer-based MAF methods on probabilistic forecasting tasks?
• RQ3 What is the impact of diffusion length (N) on predictive performance, and how does TimeGrad scale with dataset size and dimensionality?
• RQ4 What role do covariates and scale normalization play in the quality of probabilistic forecasts?

Key findings

• TimeGrad achieves state-of-the-art performance on six real-world datasets for multivariate probabilistic forecasting, outperforming a wide range of baselines on CRPS_sum.
• Across datasets, TimeGrad often yields the best CRPS_sum scores, with TimeGradRow entries in Table 2 showing competitive or superior results.
• The ablation study shows diffusion length N can be reduced to around 10 with minimal loss, with an optimal around N ≈ 100 for Electricity.
• TimeGrad demonstrates strong performance even with very high-dimensional outputs (up to 963 dimensions in Traffic).
• Ablations indicate the model benefits from autoregressive conditioning and diffusion-based emission modeling without requiring flow-based transformations during training.