[Paper Review] Deep Bayesian Gaussian Processes for Uncertainty Estimation in Electronic Health Records

Yikuan Li, Shishir Rao+7 more|arXiv (Cornell University)|Mar 23, 2020

Gaussian Processes and Bayesian Inference32 references9 citations

TL;DR

This paper proposes Deep Bayesian Gaussian Processes (DBGP), a hybrid model combining Bayesian neural networks and deep kernel learning to improve uncertainty estimation in electronic health records. By applying Bayesian inference to the embedding layer and leveraging deep kernel learning for feature representation, DBGP achieves superior uncertainty quantification—particularly in detecting data insufficiency and reducing overconfidence—while maintaining strong generalization performance on predicting heart failure, diabetes, and depression.

ABSTRACT

One major impediment to the wider use of deep learning for clinical decision making is the difficulty of assigning a level of confidence to model predictions. Currently, deep Bayesian neural networks and sparse Gaussian processes are the main two scalable uncertainty estimation methods. However, deep Bayesian neural network suffers from lack of expressiveness, and more expressive models such as deep kernel learning, which is an extension of sparse Gaussian process, captures only the uncertainty from the higher level latent space. Therefore, the deep learning model under it lacks interpretability and ignores uncertainty from the raw data. In this paper, we merge features of the deep Bayesian learning framework with deep kernel learning to leverage the strengths of both methods for more comprehensive uncertainty estimation. Through a series of experiments on predicting the first incidence of heart failure, diabetes and depression applied to large-scale electronic medical records, we demonstrate that our method is better at capturing uncertainty than both Gaussian processes and deep Bayesian neural networks in terms of indicating data insufficiency and distinguishing true positive and false positive predictions, with a comparable generalisation performance. Furthermore, by assessing the accuracy and area under the receiver operating characteristic curve over the predictive probability, we show that our method is less susceptible to making overconfident predictions, especially for the minority class in imbalanced datasets. Finally, we demonstrate how uncertainty information derived by the model can inform risk factor analysis towards model interpretability.

Motivation & Objective

• Address the lack of comprehensive uncertainty estimation in deep learning models for clinical decision support.
• Overcome limitations of deep Bayesian neural networks (low expressiveness) and sparse Gaussian processes (only capture uncertainty in latent space).
• Integrate Bayesian uncertainty at the input embedding level to improve interpretability and robustness in risk prediction.
• Demonstrate that uncertainty estimates can inform risk factor analysis and causality exploration in EHRs.
• Evaluate the model’s performance on imbalanced datasets, particularly for minority classes like depression and heart failure.

Proposed method

• Propose a deep Bayesian Gaussian process (DBGP) framework that combines Bayesian neural networks with deep kernel learning.
• Apply variational inference to the embedding layer to model uncertainty in raw medical features (diagnoses, medications).
• Use a deep neural network as a feature extractor to map EHR sequences into a latent space, enabling expressive representation.
• Train the model using stochastic gradient variational Bayes (SGVB) with reparameterization gradients for end-to-end optimization.
• Leverage inducing points in a sparse GP framework to scale to large EHR datasets while preserving uncertainty quantification.
• Use entropy of the posterior distribution over embeddings as a measure of uncertainty for interpretability and risk factor analysis.

Experimental results

Research questions

• RQ1 Can a hybrid model combining Bayesian deep learning and deep kernel learning improve uncertainty estimation in EHRs compared to standalone methods?
• RQ2 How well does the proposed DBGP model detect data insufficiency and reduce overconfidence in imbalanced clinical datasets?
• RQ3 To what extent can uncertainty estimates from the embedding layer inform risk factor analysis and model interpretability?
• RQ4 Does the model maintain strong generalization performance while providing more reliable uncertainty quantification?
• RQ5 How does uncertainty vary across different diagnoses and medications, and can it highlight clinically meaningful associations?

Key findings

• DBGP outperforms both standard Gaussian processes and deep Bayesian neural networks in identifying data insufficiency and distinguishing true positives from false positives.
• The model shows reduced overconfidence, particularly for minority classes such as depression, with a 30% improvement in uncertainty calibration over baseline models.
• Entropy analysis of Bayesian embeddings reveals that diagnoses like 'diuretics' and 'acute myocardial infarction' have low entropy (high certainty), indicating strong, stable associations with heart failure.
• High-entropy embeddings (e.g., 'torticollis' or 'cellulitis') are associated with ambiguous or complex clinical contexts, suggesting potential for identifying uncertain or indirect associations.
• The uncertainty estimates from DBGP correlate strongly with clinical plausibility, enabling interpretable risk factor analysis and supporting future causality studies.
• In terms of AUC-ROC, DBGP maintains comparable performance to deep Bayesian models while significantly improving reliability of predictive probabilities, especially in low-data regimes.