[Paper Review] Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows

Kashif Rasul, Abdul-Saboor Sheikh+3 more|arXiv (Cornell University)|Feb 14, 2020

Time Series Analysis and Forecasting46 references55 citations

TL;DR

The paper proposes an autoregressive conditional normalizing flow model (Real NVP/MAF) conditioned on RNN/Transformer states to forecast high-dimensional multivariate time series with uncertainty, achieving state-of-the-art results on several datasets.

ABSTRACT

Time series forecasting is often fundamental to scientific and engineering problems and enables decision making. With ever increasing data set sizes, a trivial solution to scale up predictions is to assume independence between interacting time series. However, modeling statistical dependencies can improve accuracy and enable analysis of interaction effects. Deep learning methods are well suited for this problem, but multivariate models often assume a simple parametric distribution and do not scale to high dimensions. In this work we model the multivariate temporal dynamics of time series via an autoregressive deep learning model, where the data distribution is represented by a conditioned normalizing flow. This combination retains the power of autoregressive models, such as good performance in extrapolation into the future, with the flexibility of flows as a general purpose high-dimensional distribution model, while remaining computationally tractable. We show that it improves over the state-of-the-art for standard metrics on many real-world data sets with several thousand interacting time-series.

Motivation & Objective

• Motivate probabilistic, multivariate time series forecasting beyond univariate or independent assumptions.
• Develop an end-to-end trainable model that captures dependencies between thousands of interacting time series.
• Combine autoregressive dynamics with flexible density estimation via conditioned normalizing flows.
• Demonstrate scalability and competitive performance on real-world datasets.
• Explore conditioning mechanisms via RNNs and Transformers to capture temporal structure.

Proposed method

• Model the conditional joint distribution p(X_t|h_t) with a flow (Real NVP or MAF) conditioned on an autoregressive state h_t.
• Use an autoregressive backbone (RNN or Transformer) to generate conditioning for the flow.
• Train by maximizing the conditional log-likelihood across time steps and series using SGD/Adam.
• Apply normalization (per-series mean scaling) to stabilize training and improve performance.
• Handle conditioning through concatenation of h with flow inputs or via bias terms, preserving Jacobian properties.
• Inference by sampling from the flow conditioned on the learned h_t, enabling trajectory sampling and uncertainty quantification.

Experimental results

Research questions

• RQ1 Can conditioned normalizing flows model the full joint distribution of high-dimensional time series efficiently?
• RQ2 Do autoregressive conditioners (RNNs or Transformers) enable scalable, accurate probabilistic forecasting for thousands of interacting series?
• RQ3 How does conditioning affect the ability to capture cross-series dependencies and temporal dynamics?
• RQ4 What is the empirical performance (CRPS) of flow-based multivariate forecasts on real-world benchmarks compared to state-of-the-art methods?

Key findings

• The proposed LSTM-Real-NVP, LSTM-MAF, and Transformer-MAF achieve state-of-the-art CRPS_sum on six real-world datasets.
• Transformer-MAF and RNN-conditioned flows outperform competitive baselines including Vec-LSTM and GP-based methods on multiple metrics.
• Cross-covariance structures observed in data are largely recovered by samples drawn from the model, indicating good dependency learning.
• Using flow-based emissions with autoregressive conditioning scales to thousands of interacting time series and remains computationally tractable.
• MAF generally provides better density modeling than Real-NVP in the evaluated settings.