[Paper Review] Fast Out-of-Sample Predictions for Bayesian Hierarchical Models of Latent Health States

Aaron Fisher, R. Yates Coley+1 more|arXiv (Cornell University)|Oct 29, 2015

Statistical Methods and Inference10 references3 citations

TL;DR

This paper proposes using importance sampling (IS) to enable fast, real-time out-of-sample predictions of latent health states in Bayesian hierarchical models, specifically for prostate cancer risk assessment. By leveraging pre-computed population-level posteriors, IS achieves high agreement with slow batch MCMC—within 1.3% RMSD—while reducing computation time from hours to 1–10 seconds per patient, enabling in-clinic decision support.

ABSTRACT

Hierarchical Bayesian models can be especially useful in precision medicine settings, where clinicians are interested in estimating the patient-level latent variables associated with an individual's current health state and its trajectory. Such models are often fit using batch Markov Chain Monte Carlo (MCMC). However, the slow speed of batch MCMC computation makes it difficult to implement in clinical settings, where immediate latent variable estimates are often desired in response to new patient data. In this report, we discuss how importance sampling (IS) can instead be used to obtain fast, in-clinic estimates of patient-level latent variables. We apply IS to the hierarchical model proposed in Coley et al (2015) for predicting an individual's underlying prostate cancer state. We find that latent variable estimates via IS can typically be obtained in 1-10 seconds per person and have high agreement with estimates coming from longer-running batch MCMC methods. Alternative options for out-of-sample fitting and online updating are also discussed.

Motivation & Objective

• To address the computational bottleneck of batch MCMC in clinical settings where real-time patient-level latent variable estimates are needed.
• To develop a scalable, low-latency method for updating individual health state predictions when new patient data arrives.
• To enable in-clinic implementation of precision medicine models without requiring repeated full model re-fitting or cross-server communication.
• To evaluate the accuracy and efficiency of importance sampling as a substitute for MCMC in out-of-sample prediction for latent health states.
• To combine IS with periodic MCMC re-fitting to maintain population-level parameter accuracy while enabling fast individual predictions.

Proposed method

• Apply importance sampling to generate posterior estimates of patient-level latent variables (e.g., aggressive vs. indolent prostate cancer state) using only the population-level parameter posteriors from prior MCMC runs.
• Use a proposal distribution $ g(b_i | \theta^{(j)}) $ centered on posterior draws $ \theta^{(j)} $ to generate candidate values for individual-level parameters $ b_i $.
• Generate 5 million proposals per patient by drawing 10 values per posterior draw across 500,000 posterior samples, increasing proposal diversity.
• Weight the proposals using unnormalized posterior densities to compute effective sample sizes and estimate posterior probabilities of latent states.
• Implement a dynamic proposal strategy that increases proposal count until effective sample size exceeds 1,000, balancing accuracy and speed.
• Compare IS-based risk estimates to those from full batch MCMC to assess agreement and computational efficiency.

Experimental results

Research questions

• RQ1 Can importance sampling produce accurate out-of-sample estimates of latent health states in Bayesian hierarchical models with minimal computational overhead?
• RQ2 How does the accuracy of IS-based predictions compare to batch MCMC estimates in terms of root mean square deviation and maximum absolute difference?
• RQ3 What is the trade-off between computational speed and estimation accuracy when varying the number of proposal samples in IS?
• RQ4 Can IS be effectively combined with periodic MCMC re-fitting to maintain long-term model accuracy while enabling real-time predictions?
• RQ5 How does the effective sample size in IS relate to prediction error, and does it follow theoretical expectations?

Key findings

• The dynamic importance sampling approach achieved a root mean square difference (rMSD) of 1.3% between IS and MCMC estimates on the probability scale, indicating high agreement.
• The maximum absolute difference between IS and MCMC estimates was 5.9%, with 99% of patients having a difference below 4.5%.
• With fixed proposal counts, the 50,000-proposal method had an rMSD of 1.6% and maximum difference of 16.6%, while the 5 million-proposal method achieved rMSD of 1.3% and maximum difference of 4.8%.
• Computation time per patient ranged from 1.4 to 2.8 seconds for the most efficient IS approaches, compared to hours for full MCMC re-fitting.
• The dynamic proposal method achieved high accuracy with median computation time of 2.3–4.6 seconds per patient, outperforming fixed 50,000-proposal method in speed and accuracy.
• Rejection sampling performed worse than IS, with an rMSD of 1.6%, suggesting IS is more effective for this application.