[Paper Review] Disease phenotyping using deep learning: A diabetes case study

Sina Rashidian, Janos Hajagos+10 more|arXiv (Cornell University)|Nov 28, 2018

Machine Learning in Healthcare10 references4 citations

TL;DR

This study proposes a deep learning model that mimics clinical coders' decisions to improve diabetes phenotyping using EHR data, leveraging demographics, lab results, medications, and prior diagnosis codes. The model achieved high accuracy in predicting ICD codes and revealed that 9.07% of diabetes cases had incorrect or missing codes, with expert review confirming the model outperformed coders in high-confidence disagreement cases.

ABSTRACT

Characterization of a patient clinical phenotype is central to biomedical informatics. ICD codes, assigned to inpatient encounters by coders, is important for population health and cohort discovery when clinical information is limited. While ICD codes are assigned to patients by professionals trained and certified in coding there is substantial variability in coding. We present a methodology that uses deep learning methods to model coder decision making and that predicts ICD codes. Our approach predicts codes based on demographics, lab results, and medications, as well as codes from previous encounters. We are able to predict existing codes with high accuracy for all three of the test cases we investigated: diabetes, acute renal failure, and chronic kidney disease. We employed a panel of clinicians, in a blinded manner, to assess ground truth and compared the predictions of coders, model and clinicians. When disparities between the model prediction and coder assigned codes were reviewed, our model outperformed coder assigned ICD codes.

Motivation & Objective

• To address inaccuracies in ICD coding for diabetes by modeling coder decision-making using deep learning.
• To develop a generalizable, data-driven method for disease phenotyping without requiring disease-specific feature engineering.
• To evaluate model performance against expert physician review in discordant cases to assess accuracy and reliability.
• To estimate global miscoding rates for diabetes in EHRs using model predictions and expert validation.
• To test the scalability and generalizability of the model across multiple healthcare facilities.

Proposed method

• A multilayer perceptron deep neural network was trained on de-identified EHR data mapped to the OHDSI Common Data Model, using demographic, lab, medication, and prior diagnosis code features.
• Features were filtered to include only those present in at least 5% of positive diabetes cases, reducing dimensionality from 13,139 to 966 to reduce sparsity.
• Prior encounter diagnosis codes were aggregated into a binary vector to represent historical disease burden, enabling the model to learn temporal patterns without requiring RNNs.
• Model outputs used a sigmoid activation to produce a probability score (p) representing the likelihood of diabetes, which was used to stratify disagreement cases by confidence level.
• Expert physicians reviewed 120 discordant cases in a double-blind manner, assessing both diagnosis presence and their own confidence in the assessment.
• The model was retrained on data from 11 facilities to evaluate generalizability, with performance assessed on independent test sets from each facility.

Experimental results

Research questions

• RQ1 Can a deep learning model accurately reproduce ICD coding decisions for diabetes using standard EHR data elements?
• RQ2 How does the model’s prediction confidence correlate with actual clinical likelihood of diabetes as assessed by expert physicians?
• RQ3 What proportion of diabetes coding errors exist in real-world EHR data, and how do they break down between missing and incorrect codes?
• RQ4 Can a single deep learning model trained on multi-facility data generalize effectively across different healthcare institutions?
• RQ5 To what extent does the model’s disagreement with coders reflect actual coding errors, as validated by expert review?

Key findings

• The deep learning model achieved an F1-score of 0.80 when trained and tested on data from a single facility, indicating strong performance in predicting diabetes ICD codes.
• In cases of high-confidence disagreement, 35 out of 40 expert reviews agreed with the model, demonstrating that coders were significantly more likely to be wrong when the model was highly confident.
• The model predicted that 9.07% of all encounters had incorrect or missing diabetes codes, with 5.68% missing and 3.39% incorrectly coded.
• The model’s predicted probability of diabetes closely tracked the physician-established likelihood of disease, validating its calibration across confidence intervals.
• When trained on data from 11 facilities, the model achieved an F1-score of 0.79 on facility 143, showing strong generalization and robustness to facility-specific batch effects.
• Expert review confirmed that the model’s confidence scores accurately reflected the true prevalence of diabetes in each disagreement group, indicating reliable uncertainty estimation.