[Paper Review] A Meta-embedding-based Ensemble Approach for ICD Coding Prediction

Pavithra Rajendran, Alexandros Zenonos+2 more|arXiv (Cornell University)|Feb 26, 2021

Topic Modeling19 references4 citations

TL;DR

This paper proposes a meta-embedding-based ensemble model that improves ICD coding prediction by fusing multiple pretrained word embeddings—derived from clinical notes and external biomedical sources—using geometric post-processing and locally linear meta-embedding techniques. The approach enhances performance on both ICD-9 and ICD-10 coding tasks without architectural complexity, outperforming baselines by preserving local vector neighborhoods during fusion.

ABSTRACT

International Classification of Diseases (ICD) are the de facto codes used globally for clinical coding. These codes enable healthcare providers to claim reimbursement and facilitate efficient storage and retrieval of diagnostic information. The problem of automatically assigning ICD codes has been approached in literature as a multilabel classification, using neural models on unstructured data. Our proposed approach enhances the performance of neural models by effectively training word vectors using routine medical data as well as external knowledge from scientific articles. Furthermore, we exploit the geometric properties of the two sets of word vectors and combine them into a common dimensional space, using meta-embedding techniques. We demonstrate the efficacy of this approach for a multimodal setting, using unstructured and structured information. We empirically show that our approach improves the current state-of-the-art deep learning architectures and benefits ensemble models.

Motivation & Objective

• To improve multilabel ICD code prediction for clinical notes using multimodal data (unstructured text and structured vitals).
• To enhance word embeddings from clinical summaries by integrating external biomedical knowledge without increasing dimensionality.
• To investigate whether geometric fusion of diverse pretrained embeddings improves ICD coding performance.
• To develop an ensemble model that combines predictions from multiple modalities, including text and structured data, for superior classification.

Proposed method

• Post-process pretrained word embeddings (MIMIC and PubMed) using MeanDiff to align semantic spaces while preserving local neighborhood structures.
• Apply locally linear meta-embedding techniques to combine multiple embedding sets into a unified, lower-dimensional representation.
• Train separate deep learning models (e.g., CNN) on different modalities: unstructured text (discharge summaries) and structured data (vitals).
• Ensemble predictions from modality-specific models using a meta-classifier to improve final multilabel classification performance.
• Use ICD code definitions as external knowledge to enrich label representation, improving semantic alignment with clinical text.
• Evaluate performance on top 32 ICD-10 and 50 ICD-9 codes to manage class imbalance and ensure reliable evaluation.

Experimental results

Research questions

• RQ1 Can combining multiple pretrained word embeddings using geometric meta-embedding techniques improve ICD code prediction beyond single-embedding baselines?
• RQ2 Does post-processing embeddings to preserve local neighborhood structure enhance downstream classification performance?
• RQ3 Can multimodal fusion of unstructured text and structured clinical data yield better ICD coding results than unimodal approaches?
• RQ4 Is the proposed embedding fusion method effective across different ICD coding systems (ICD-9 vs ICD-10)?
• RQ5 Does integrating external biomedical knowledge into word embeddings improve model generalization without increasing model complexity?

Key findings

• The locally linear meta-embedding technique with MeanDiff post-processing of MIMIC and PubMed embeddings achieved the highest performance across all evaluated configurations.
• The best-performing model outperformed baseline methods on both ICD-10 (top 32 codes) and ICD-9 (top 50 codes) multilabel classification tasks.
• Ensembling structured data predictions with a CNN model using fused embeddings significantly improved performance compared to using structured data alone.
• The highest-performing configuration was consistent across both ICD-9 and ICD-10 coding tasks, indicating robustness to coding system differences.
• The use of external knowledge via embedding fusion enhanced model performance without requiring architectural changes or additional training data.