[Paper Review] Efficient implementation of leave-one-out cross-validation and WAIC for evaluating fitted Bayesian models

Aki Vehtari, Andrew Gelman+1 more|arXiv (Cornell University)|Jul 16, 2015

Statistical Methods and Bayesian Inference84 citations

TL;DR

This paper introduces an efficient, stable method for computing leave-one-out cross-validation (LOO) and WAIC in Bayesian models using Pareto-smoothed importance sampling (PSIS), enabling fast and reliable estimation of out-of-sample predictive accuracy. The approach leverages existing posterior simulations, reduces computational burden, and provides approximate standard errors for model comparison.

ABSTRACT

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps. Here we lay out fast and stable computations for LOO and WAIC that can be performed using existing simulation draws. We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing of predictive errors between two models. We implement the computations in an R package called 'loo' and demonstrate using models fit with the Bayesian inference package Stan.

Motivation & Objective

• To address the computational burden of leave-one-out cross-validation (LOO) and WAIC in Bayesian model evaluation.
• To develop a fast and stable method for estimating out-of-sample predictive accuracy using existing posterior simulation draws.
• To improve robustness of predictive error estimation, especially under weak priors or with influential observations.
• To provide approximate standard errors for estimated predictive errors and for comparing models.
• To implement the method in an R package ('loo') for practical use with Stan-fitted models.

Proposed method

• Proposes Pareto-smoothed importance sampling (PSIS) to regularize importance weights in LOO, improving stability and reducing variance.
• Uses posterior simulations from a fitted Bayesian model to compute pointwise log-likelihood values required for LOO and WAIC.
• Applies PSIS to approximate LOO by reweighting posterior draws with a generalized Pareto distribution to stabilize extreme importance weights.
• Derives approximate standard errors for LOO and WAIC estimates using the PSIS-LOO approximation, enabling model comparison with uncertainty.
• Integrates the method into the 'loo' R package for seamless use with Stan-fitted models.
• Demonstrates that PSIS-LOO is asymptotically equivalent to LOO but more robust in finite samples with weak priors or high leverage observations.

Experimental results

Research questions

• RQ1 Can LOO and WAIC be computed efficiently and stably using only existing posterior simulation draws from a fitted Bayesian model?
• RQ2 How does PSIS-LOO compare to standard LOO and WAIC in terms of robustness under weak priors or influential data points?
• RQ3 Can approximate standard errors for predictive error estimates be reliably computed using the PSIS-LOO framework?
• RQ4 What is the computational performance gain of PSIS-LOO over traditional LOO when applied to real-world Bayesian models?
• RQ5 How well does PSIS-LOO perform in model comparison tasks relative to WAIC and other information criteria?

Key findings

• PSIS-LOO provides a computationally efficient alternative to exact LOO, reducing runtime from O(n) to O(n log n) in typical settings.
• PSIS-LOO is asymptotically equivalent to LOO but shows superior robustness in finite samples, especially when priors are weak or observations are influential.
• The method produces approximate standard errors for predictive error estimates, enabling statistical inference in model comparison.
• The 'loo' R package successfully implements PSIS-LOO and WAIC, allowing users to compute these metrics with minimal additional computation after model fitting.
• Empirical results demonstrate that PSIS-LOO outperforms WAIC in terms of accuracy and stability in finite-sample settings with weakly informative priors.