[Paper Review] All Data Inclusive, Deep Learning Models to Predict Critical Events in the Medical Information Mart for Intensive Care III Database (MIMIC III)

Anubhav Reddy Nallabasannagari, Madhu Reddiboina+4 more|arXiv (Cornell University)|Sep 2, 2020

Machine Learning in Healthcare32 references8 citations

TL;DR

This study develops deep learning models using all available data sources from the MIMIC-III ICU database—structured and unstructured—via natural language processing to predict in-hospital mortality (IHM) and length of stay ≥7 days. Models trained on all data sources (AS) achieved AUC-ROC of 0.9178 for IHM and 0.8806 for LOS, significantly outperforming models using only chart data (CS), demonstrating that comprehensive EHR integration enhances predictive accuracy for critical care outcomes.

ABSTRACT

Intensive care clinicians need reliable clinical practice tools to preempt unexpected critical events that might harm their patients in intensive care units (ICU), to pre-plan timely interventions, and to keep the patient's family well informed. The conventional statistical models are built by curating only a limited number of key variables, which means a vast unknown amount of potentially precious data remains unused. Deep learning models (DLMs) can be leveraged to learn from large complex datasets and construct predictive clinical tools. This retrospective study was performed using 42,818 hospital admissions involving 35,348 patients, which is a subset of the MIMIC-III dataset. Natural language processing (NLP) techniques were applied to build DLMs to predict in-hospital mortality (IHM) and length of stay >=7 days (LOS). Over 75 million events across multiple data sources were processed, resulting in over 355 million tokens. DLMs for predicting IHM using data from all sources (AS) and chart data (CS) achieved an AUC-ROC of 0.9178 and 0.9029, respectively, and PR-AUC of 0.6251 and 0.5701, respectively. DLMs for predicting LOS using AS and CS achieved an AUC-ROC of 0.8806 and 0.8642, respectively, and PR-AUC of 0.6821 and 0.6575, respectively. The observed AUC-ROC difference between models was found to be significant for both IHM and LOS at p=0.05. The observed PR-AUC difference between the models was found to be significant for IHM and statistically insignificant for LOS at p=0.05. In this study, deep learning models were constructed using data combined from a variety of sources in Electronic Health Records (EHRs) such as chart data, input and output events, laboratory values, microbiology events, procedures, notes, and prescriptions. It is possible to predict in-hospital mortality with much better confidence and higher reliability from models built using all sources of data.

Motivation & Objective

• To develop deep learning models that leverage all available structured and unstructured data from electronic health records (EHRs) for predicting critical ICU outcomes.
• To evaluate whether models trained on all data sources (AS) outperform models trained only on chart events (CS) in predicting in-hospital mortality (IHM) and length of stay ≥7 days (LOS).
• To demonstrate the value of incorporating unstructured clinical notes, lab results, procedures, and medication data via NLP for improved predictive performance.
• To provide a data engineering pipeline that integrates heterogeneous EHR data sources without hand-picked features or fixed time sampling.
• To establish a foundation for next-generation clinical decision support tools using comprehensive, real-world EHR data.

Proposed method

• The study uses the MIMIC-III v1.4 dataset, comprising 42,818 hospital admissions from Beth Israel Deaconess Medical Center.
• All data sources—chart events, input/output, lab results, microbiology, procedures, notes, and prescriptions—were combined into a single data stream for model training.
• Natural language processing (NLP) techniques were applied to tokenize free-text clinical notes into unigrams, preserving temporal event sequences.
• Deep learning models were trained end-to-end on raw event sequences without feature engineering or fixed temporal binning.
• Two model variants were compared: one using all data sources (AS) and one using only chart events (CS) for each outcome.
• Performance was evaluated using AUC-ROC and PR-AUC, with statistical significance assessed at p=0.05.

Experimental results

Research questions

• RQ1 Can deep learning models trained on all available EHR data sources predict in-hospital mortality more accurately than models using only chart data?
• RQ2 Does incorporating unstructured clinical notes and diverse data types improve prediction performance for length of stay ≥7 days?
• RQ3 Is the performance gain from using all data sources statistically significant for both IHM and LOS outcomes?
• RQ4 How does the absence of temporal structure in tokenized sequences affect model performance, and what are the implications for future modeling?
• RQ5 To what extent can NLP-based integration of heterogeneous EHR data enhance predictive reliability in critical care?

Key findings

• The deep learning model trained on all data sources (AS) achieved an AUC-ROC of 0.9178 for in-hospital mortality, significantly outperforming the chart-only model (CS) with AUC-ROC of 0.9029 (p=0.05).
• For in-hospital mortality, the AS model achieved a PR-AUC of 0.6251, compared to 0.5701 for the CS model, with a statistically significant difference (p=0.05).
• For length of stay ≥7 days, the AS model achieved an AUC-ROC of 0.8806, compared to 0.8642 for the CS model, with a significant difference (p=0.05).
• The PR-AUC for LOS showed a difference of 0.6821 (AS) vs. 0.6575 (CS), which was not statistically significant (p=0.05).
• Over 75 million events from 35,348 patients were processed, resulting in over 355 million tokens, demonstrating scalability of the data pipeline.
• The study confirms that comprehensive EHR integration via NLP significantly improves predictive performance, especially for mortality prediction, without relying on hand-selected features.