[Paper Review] Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Hejie Cui, Xinyu Fang+4 more|arXiv (Cornell University)|Feb 19, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes MINGLE, a multimodal fusion framework that integrates structured EHR data and clinical notes using a two-level semantics infusion strategy within a hypergraph neural network. By leveraging large language models (LLMs) to enrich medical concept semantics and clinical note semantics, MINGLE improves visit-level representation learning, achieving a 11.83% relative performance gain on two EHR datasets compared to baselines.

ABSTRACT

Electronic Health Records (EHRs) have become increasingly popular to support clinical decision-making and healthcare in recent decades. EHRs usually contain heterogeneous information, such as structural data in tabular form and unstructured data in textual notes. Different types of information in EHRs can complement each other and provide a more complete picture of the health status of a patient. While there has been a lot of research on representation learning of structured EHR data, the fusion of different types of EHR data (multimodal fusion) is not well studied. This is mostly because of the complex medical coding systems used and the noise and redundancy present in the written notes. In this work, we propose a new framework called MINGLE, which integrates both structures and semantics in EHR effectively. Our framework uses a two-level infusion strategy to combine medical concept semantics and clinical note semantics into hypergraph neural networks, which learn the complex interactions between different types of data to generate visit representations for downstream prediction. Experiment results on two EHR datasets, the public MIMIC-III and private CRADLE, show that MINGLE can effectively improve predictive performance by 11.83% relatively, enhancing semantic integration as well as multimodal fusion for structural and textual EHR data.

Motivation & Objective

• To address the challenge of multimodal fusion in EHRs by integrating heterogeneous structured data and unstructured clinical notes.
• To overcome limitations in existing methods that treat EHR data in isolation, especially the lack of semantic integration for medical codes and clinical notes.
• To enhance representation learning of patient visits by fusing medical concept semantics and clinical note semantics into a hypergraph-based model.
• To leverage the contextual understanding and domain knowledge of large language models (LLMs) for improved semantic representation in clinical prediction tasks.
• To develop a unified framework that models complex, higher-order interactions in EHRs while incorporating rich semantic signals from both structured codes and textual notes.

Proposed method

• MINGLE employs a hypergraph neural network as the backbone to model higher-order interactions among medical codes and patient visits.
• It introduces a two-level semantics infusion strategy: first infusing medical concept name semantics from LLM-generated embeddings into the hypergraph structure, then fusing clinical note semantics via LLM-based encoding.
• Medical concept semantics are derived from LLMs by prompting with concept names (e.g., 'Angiotensin II receptor blocker') to generate contextual embeddings.
• Clinical note semantics are encoded using LLMs to extract salient information, which are then fused into the hypergraph representation using a learnable attention mechanism.
• The framework jointly optimizes structural and semantic representations through a contrastive learning objective to align semantic and structural views.
• A dimension ratio hyperparameter controls the balance between structural and semantic embedding contributions in the final representation.

Experimental results

Research questions

• RQ1 Can the integration of medical concept semantics from LLMs improve the representation of structured EHR data in hypergraph models?
• RQ2 How does the fusion of clinical note semantics via LLMs affect the predictive performance of multimodal EHR models?
• RQ3 Does the two-level semantics infusion strategy lead to more robust and clinically interpretable patient visit representations?
• RQ4 How does MINGLE compare to existing baselines in terms of performance across multiple metrics on real-world EHR datasets?
• RQ5 To what extent do LLM-derived semantics enhance the identification of clinically relevant nodes in EHR hypergraphs?

Key findings

• MINGLE achieves a 11.83% relative improvement in predictive performance over baselines on two EHR datasets, MIMIC-III and CRADLE.
• On MIMIC-III, MINGLE achieves the highest F1 score among all baselines, demonstrating superior performance on imbalanced data.
• Ablation studies show that removing medical concept semantics leads to a significant performance drop, indicating its critical role in modeling structured EHR data.
• The influence of clinical note semantics is more pronounced on the CRADLE dataset, which contains cleaner, more medical-concept-focused notes, suggesting context quality affects fusion effectiveness.
• Case studies reveal that MINGLE identifies clinically relevant disease nodes such as 'Heart Failure' and 'Cardiac Dysrhythmias' that baseline models miss, improving predictive reasoning.
• MINGLE successfully captures key clinical information from notes—such as post-surgical arrhythmias and medication side effects—demonstrating enhanced contextual understanding.