[Paper Review] Accurate Computation of Marginal Data Densities Using Variational Bayes

Gholamreza Hajargasht, Tomasz Woźniak|arXiv (Cornell University)|May 25, 2018

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a novel marginal data density estimator (MDDE) that leverages variational Bayes (VB) posterior approximations as importance sampling weights in reciprocal importance sampling (RIS), enabling accurate, stable, and computationally efficient estimation of marginal likelihoods. The method achieves lower numerical standard errors and bias than existing MDDEs, with finite variance and asymptotic normality, while avoiding truncation of the weighting density.

ABSTRACT

We propose a new marginal data density estimator (MDDE) that uses the variational Bayes posterior density as a weighting density of the reciprocal importance sampling (RIS) MDDE. This computationally convenient estimator is based on variational Bayes posterior densities that are available for many models and requires simulated draws only from the posterior distribution. It provides accurate estimates with a moderate number of posterior draws, has a finite variance, and provides a minimum variance candidate for the class of RIS MDDEs. Its reciprocal is consistent, asymptotically normally distributed, and unbiased. These properties are obtained without truncating the weighting density, which is typical for other such estimators. Our proposed estimators outperform many existing MDDEs in terms of bias and numerical standard errors. In particular, our RIS MDDE performs uniformly better than other estimators from this class.

Motivation & Objective

• To develop a computationally efficient and accurate marginal data density estimator (MDDE) for Bayesian model comparison.
• To overcome limitations of existing MDDEs, such as high bias, infinite variance, and reliance on truncated weighting densities.
• To leverage the variational Bayes (VB) posterior as a natural, analytically tractable importance sampling density for RIS-based MDDEs.
• To demonstrate that VB-based MDDEs achieve minimum variance among RIS estimators and maintain consistency and asymptotic normality.
• To extend the applicability of MDDEs to models where MCMC sampling is complex or infeasible, by relying only on posterior draws and VB approximations.

Proposed method

• The proposed RIS MDDE uses the VB posterior density as the importance sampling weighting function, avoiding truncation of the support.
• The estimator is defined as the reciprocal of the average of the inverse ratio between the VB density and the joint likelihood-prior product over posterior draws.
• The method relies on posterior samples and the VB approximation, which is available for many models without requiring full MCMC sampling.
• Theoretical properties include consistency of the reciprocal estimator, asymptotic normality, and unbiasedness, all without truncation.
• The approach is extended to Bridge Sampling (BS) MDDEs using the same VB posterior as the proposal density.
• The method is applicable beyond mean-field VB, as it does not require the factorization assumption in the VB approximation.

Experimental results

Research questions

• RQ1 Can variational Bayes posterior approximations be used effectively as importance sampling weights to estimate marginal data densities?
• RQ2 Does using VB posteriors as weighting densities yield MDDEs with lower bias and numerical standard errors than existing methods?
• RQ3 Can the resulting RIS MDDE achieve finite variance and asymptotic normality without truncating the weighting density?
• RQ4 How does the performance of VB-based MDDEs compare to benchmark methods like Chib’s and Geweke’s in complex models?
• RQ5 To what extent does the VB approximation’s optimality in minimizing reverse KL divergence translate into improved MDDE efficiency?

Key findings

• The proposed RIS MDDE based on VB posterior densities achieves the lowest numerical standard error (NSE) among all estimators tested, with NSE values nearly fifteen times lower than Geweke’s (1999) RIS MDDE.
• The VB-based RIS and BS MDDEs outperform Chib’s (1995) benchmark MDDE and all other competitors in terms of both bias and NSE across multiple models, including stochastic frontier and longitudinal Poisson models.
• The reciprocal of the RIS MDDE is consistent, asymptotically normal, and unbiased, even without truncating the weighting density, which is a key advantage over prior methods.
• In models with non-standard posterior computation (e.g., Metropolis-Hastings within Gibbs), the VB RIS MDDE remains numerically stable and precise, with NSEs significantly lower than alternative estimators.
• The VB-based estimators show superior numerical stability, as evidenced by box plots showing tightly clustered estimates around the true value, unlike other estimators with high variability.
• The method maintains high performance even when the complete data likelihood is used, with only a moderate (around twofold) increase in NSE, unlike previous studies that reported severe efficiency losses.