[Paper Review] Gaussian variational approximation for high-dimensional state space models

Matias Quiroz, David J. Nott+1 more|arXiv (Cornell University)|Jan 24, 2018

Gaussian Processes and Bayesian Inference60 references18 citations

TL;DR

This paper proposes a Gaussian variational approximation for high-dimensional state space models by leveraging a dynamic factor model to parsimoniously parameterize the variational posterior covariance matrix. By reducing the state dimension and exploiting Markovian temporal dependence, the method enables efficient stochastic gradient optimization and delivers accurate predictive inference in two high-dimensional applications: spatio-temporal modeling of bird migration and multivariate stochastic volatility in finance.

ABSTRACT

Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number of model parameters, so it is necessary to find simple yet effective parameterizations of the covariance structure when the number of model parameters is large. We approximate the joint posterior distribution over the high-dimensional state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. The usefulness of the approach is illustrated for prediction in two high-dimensional applications that are challenging for Markov chain Monte Carlo sampling. The first is a spatio-temporal model for the spread of the Eurasian Collared-Dove across North America; the second is a Wishart-based multivariate stochastic volatility model for financial returns.

Motivation & Objective

• Address the computational infeasibility of full covariance parameterization in high-dimensional state space models by reducing the number of variational parameters.
• Develop a scalable variational inference method that maintains accurate uncertainty quantification and predictive performance despite model complexity.
• Enable efficient posterior approximation in high-dimensional settings where MCMC sampling is computationally prohibitive.
• Utilize a dynamic factor model structure to capture both cross-sectional dependence and temporal correlation in the state vector.
• Demonstrate the method's effectiveness on two real-world high-dimensional problems: spatio-temporal spread modeling and multivariate stochastic volatility estimation.

Proposed method

• Parameterize the variational posterior covariance matrix using a dynamic factor model, reducing the state dimension and inducing conditional independence via low-rank structure.
• Model the factors as a first-order Markov process to induce sparsity in the precision matrix and enable efficient computation.
• Use stochastic gradient ascent with the reparameterization trick to optimize the evidence lower bound (ELBO) efficiently in high dimensions.
• Apply the Woodbury formula to accelerate matrix operations in the variational inference procedure, enabling scalability.
• Match the conditional independence structure of the true posterior by aligning the sparsity pattern of the variational precision matrix with that of the posterior.
• Use a low-dimensional mean parameterization for the variational posterior to further reduce computational burden in high-dimensional state spaces.

Experimental results

Research questions

• RQ1 Can a dynamic factor structure in the variational posterior covariance matrix effectively reduce the number of parameters while preserving predictive accuracy in high-dimensional state space models?
• RQ2 How well does the proposed Gaussian variational approximation perform in comparison to MCMC in terms of predictive density estimation for high-dimensional spatio-temporal and financial time series?
• RQ3 To what extent does the use of stochastic gradient optimization with the reparameterization trick enable scalable inference in high-dimensional settings?
• RQ4 Does the factor-based covariance parameterization capture the key dependence structures in the true posterior, particularly in models with complex temporal and spatial dependencies?
• RQ5 Can the method maintain reliable uncertainty quantification and prediction intervals even when the true posterior exhibits skewness or heavy tails?

Key findings

• The dynamic factor model parameterization significantly reduces the number of variational parameters, making inference feasible in high-dimensional state space models with up to 78 states.
• The method achieves predictive performance comparable to MCMC in both the Eurasian collared-dove spatio-temporal model and the multivariate stochastic volatility model, as confirmed by predictive density comparisons.
• For the multivariate stochastic volatility model with k=12 assets, the variational approximation successfully produced prediction intervals that contained the withheld test observations across all horizons h=1,2,3,4.
• In-sample predictions for the real data example showed that the variational predictive densities closely matched the oracle (true) posterior densities, with the observed data point falling within the predicted intervals.
• The estimated ELBO for the real data example stabilized over iterations despite higher variability, indicating convergence of the stochastic optimization procedure.
• The method demonstrated robustness across different data realizations and parameter settings, with consistent performance in both simulated and real-world datasets.