[Paper Review] Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification

Thanh-Tung Nguyen, Viktor Schlegel+5 more|arXiv (Cornell University)|Apr 27, 2023

Machine Learning in Healthcare4 citations

TL;DR

This paper introduces Mimic-IV-ICD, a new open benchmark for extreme multi-label ICD coding using the MIMIC-IV electronic health record dataset, supporting both ICD-9 and ICD-10 codes. It standardizes data preprocessing, evaluates multiple deep learning models, and establishes reproducible baselines, with MSMN outperforming PLM-ICD in low-frequency code groups due to better code description integration.

ABSTRACT

Clinical notes are assigned ICD codes - sets of codes for diagnoses and procedures. In the recent years, predictive machine learning models have been built for automatic ICD coding. However, there is a lack of widely accepted benchmarks for automated ICD coding models based on large-scale public EHR data. This paper proposes a public benchmark suite for ICD-10 coding using a large EHR dataset derived from MIMIC-IV, the most recent public EHR dataset. We implement and compare several popular methods for ICD coding prediction tasks to standardize data preprocessing and establish a comprehensive ICD coding benchmark dataset. This approach fosters reproducibility and model comparison, accelerating progress toward employing automated ICD coding in future studies. Furthermore, we create a new ICD-9 benchmark using MIMIC-IV data, providing more data points and a higher number of ICD codes than MIMIC-III. Our open-source code offers easy access to data processing steps, benchmark creation, and experiment replication for those with MIMIC-IV access, providing insights, guidance, and protocols to efficiently develop ICD coding models.

Motivation & Objective

• To address the lack of widely accepted benchmarks for automated ICD coding using large-scale public EHR data.
• To standardize data preprocessing and model evaluation pipelines for ICD coding tasks on MIMIC-IV.
• To establish a comprehensive benchmark for both ICD-9 and ICD-10 coding with higher code counts and more data points than prior benchmarks.
• To enable reproducibility and facilitate model comparison by releasing open-source code for data processing and experiment replication.
• To accelerate research in automated ICD coding by providing a standardized, publicly accessible benchmark for future studies.

Proposed method

• The authors process raw MIMIC-IV clinical notes and map them to ICD-9 and ICD-10 codes using standardized procedures.
• They implement and compare multiple deep learning models, including PLM-ICD and MSMN, for ICD coding prediction.
• The benchmark includes full-code and high-frequency code settings, with code frequency groups used to evaluate model performance across label frequency distributions.
• The method incorporates code descriptions and synonyms to improve semantic alignment between clinical notes and ICD codes, particularly in the MSMN model.
• A public, open-source codebase is released to enable data processing, benchmark creation, and experiment replication for researchers with MIMIC-IV access.
• The pipeline supports both ICD-9 and ICD-10 coding, with the ICD-10 benchmark featuring over 155,000 codes, significantly expanding on MIMIC-III.

Experimental results

Research questions

• RQ1 How do different deep learning models perform on extreme multi-label ICD coding tasks using MIMIC-IV data?
• RQ2 What is the impact of incorporating code descriptions and synonyms on model performance, especially for rare ICD codes?
• RQ3 How does model performance vary across different code frequency groups (e.g., 1–10, 11–50, 51–100, >500) in the ICD-10 coding task?
• RQ4 Can a standardized, open benchmark improve reproducibility and accelerate progress in automated ICD coding research?
• RQ5 How does the new MIMIC-IV-ICD benchmark compare to existing MIMIC-III benchmarks in terms of data volume and code coverage?

Key findings

• MSMN outperforms PLM-ICD in the 1–10, 11–50, and 51–100 code frequency groups, particularly in macro F1 scores, due to better integration of code descriptions and synonyms.
• PLM-ICD performs better than MSMN in the >500 code group, indicating superior performance of large pretrained language models on high-frequency labels.
• The macro F1 difference between MSMN and PLM-ICD is 1% in the >500 and 101–500 groups, with MSMN showing stronger performance in lower-frequency categories.
• The benchmark includes over 155,000 ICD-10 codes and more data points than MIMIC-III, significantly expanding the scale of available public ICD coding data.
• The open-source codebase enables full reproducibility of data processing and model training, supporting future enhancements and model comparisons.
• The results suggest that combining large language models with structured code metadata (e.g., descriptions, synonyms) can improve performance on rare ICD codes, pointing to a promising future direction.