[Paper Review] Interpretation of machine learning predictions for patient outcomes in electronic health records

William La Cava, Christopher R. Bauer+2 more|PubMed|Mar 14, 2019

Machine Learning in Healthcare22 references34 citations

TL;DR

The paper compares three state-of-the-art ML methods for predicting seven patient diagnoses from EHRs and evaluates various feature-importance measures, finding permutation importance most aligns with clinical interpretation when predictions are strong.

ABSTRACT

Electronic health records are an increasingly important resource for understanding the interactions between patient health, environment, and clinical decisions. In this paper we report an empirical study of predictive modeling of seven patient outcomes using three state-of-the-art machine learning methods. Our primary goal is to validate the models by interpreting the importance of predictors in the final models. Central to interpretation is the use of feature importance scores, which vary depending on the underlying methodology. In order to assess feature importance, we compared univariate statistical tests, information-theoretic measures, permutation testing, and normalized coefficients from multivariate logistic regression models. In general we found poor correlation between methods in their assessment of feature importance, even when their performance is comparable and relatively good. However, permutation tests applied to random forest and gradient boosting models showed the most agreement, and the importance scores matched the clinical interpretation most frequently.

Motivation & Objective

• Assess predictive performance of three ML methods (LR, RF, XGBoost) on multiple patient outcomes using EHR data.
• Compare model-specific and model-agnostic feature importance measures to interpret predictors.
• Evaluate correlation and interpretability of importance scores across diseases and horizons.

Proposed method

• Use Geisinger Health Records dataset of 899,128 patients from 1996-2015 with common and rare lab measures and demographics as predictors.
• Predict three outcomes with horizons of one day, six months, and one year prior to diagnosis.
• Train penalized logistic regression, random forest, and XGBoost; tune hyper-parameters with 10-fold CV; evaluate via AUROC.
• Compute internal feature importance (coefficients, Gini, etc.) and permutation importance on test data.
• Assess correlations between importance measures and interpretability across seven diseases.
• Provide code for reproducibility at http://github.com/EpistasisLab/interpret_ehr.

Experimental results

Research questions

• RQ1 How do three ML methods (LR, RF, XGBoost) perform in predicting seven patient diagnoses from EHR data across different prediction horizons?
• RQ2 How do model-specific and model-agnostic feature importance measures compare in identifying clinically meaningful predictors?
• RQ3 To what extent do importance measures agree across models and diseases, and which measure best aligns with clinical understanding?

Key findings

• XGBoost generally yields the best predictive performance, followed closely by Random Forest; Penalized Logistic Regression is weaker, especially for Alzheimer’s disease and esophageal reflux.
• Permutation importance scores from RF and XGBoost closely align with clinical interpretation and are highly correlated across models (RF permutation vs XGBoost permutation).
• Gini importance from XGBoost does not align well with expected important predictors, while permutation importance yields more intuitive results.
• Across diseases, RF permutation and XGBoost permutation scores are highly correlated (Pearson’s R2 = 0.95).
• Univariate importance correlates least with other measures, indicating it is insufficient alone for determining important factors.
• One-year horizons show high AUROC for several diseases (e.g., Diabetes with renal manifestations 0.96); predictive ability generally declines with longer horizons.