[Paper Review] ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text

Yu Han, Peikun Guo+1 more|arXiv (Cornell University)|May 26, 2024

ECG Monitoring and Analysis4 citations

TL;DR

This paper introduces ECG Semantic Integrator (ESI), a foundation model that improves ECG representation learning by combining LLM-enhanced textual descriptions with multimodal contrastive and captioning pretraining. Using a retrieval-augmented generation pipeline (CQA) to create rich, standardized ECG reports, ESI achieves state-of-the-art performance in arrhythmia detection (1.6% AUC gain) and ECG-based subject identification (3.5% improvement) over strong self-supervised baselines.

ABSTRACT

The utilization of deep learning on electrocardiogram (ECG) analysis has brought the advanced accuracy and efficiency of cardiac healthcare diagnostics. By leveraging the capabilities of deep learning in semantic understanding, especially in feature extraction and representation learning, this study introduces a new multimodal contrastive pretaining framework that aims to improve the quality and robustness of learned representations of 12-lead ECG signals. Our framework comprises two key components, including Cardio Query Assistant (CQA) and ECG Semantics Integrator(ESI). CQA integrates a retrieval-augmented generation (RAG) pipeline to leverage large language models (LLMs) and external medical knowledge to generate detailed textual descriptions of ECGs. The generated text is enriched with information about demographics and waveform patterns. ESI integrates both contrastive and captioning loss to pretrain ECG encoders for enhanced representations. We validate our approach through various downstream tasks, including arrhythmia detection and ECG-based subject identification. Our experimental results demonstrate substantial improvements over strong baselines in these tasks. These baselines encompass supervised and self-supervised learning methods, as well as prior multimodal pretraining approaches.

Motivation & Objective

• To address the lack of rich, semantically enriched textual descriptions for ECG signals in pretraining.
• To improve ECG representation learning by integrating clinical semantics from LLM-generated reports.
• To develop a multimodal pretraining framework that jointly optimizes contrastive and captioning objectives for ECG signals.
• To evaluate the robustness and generalization of the learned representations on downstream tasks like arrhythmia classification and subject identification.
• To investigate the impact of ECG-text alignment quality on model performance.

Proposed method

• A retrieval-augmented generation (RAG) pipeline, Cardio Query Assistant (CQA), generates detailed, standardized ECG reports using LLMs and external medical knowledge.
• CQA enriches ECG descriptions with demographic data, diagnostic conditions, and waveform pattern details.
• The ECG Semantics Integrator (ESI) framework uses a dual-encoder architecture to learn joint representations of ECG signals and their textual descriptions.
• ESI applies both contrastive loss (to align matching ECG-text pairs) and captioning loss (to reconstruct textual descriptions from ECG features).
• Pretraining is performed on ~650,000 12-lead ECGs with CQA-generated text, followed by linear probing for downstream evaluation.
• The framework is evaluated on PTB-XL and ICBEB arrhythmia datasets, with ablation studies on data size and ECG-text misalignment.

Experimental results

Research questions

• RQ1 Can LLM-enhanced, retrieval-augmented textual descriptions improve the quality of ECG representation learning?
• RQ2 How does joint pretraining with contrastive and captioning losses affect downstream ECG analysis performance?
• RQ3 What is the impact of pretraining data size and distribution shift on model generalization?
• RQ4 How sensitive is the model to misalignment between ECG signals and their textual descriptions?
• RQ5 Can the proposed framework outperform strong supervised and self-supervised baselines in arrhythmia detection and ECG-based subject identification?

Key findings

• The ESI model achieves a 1.6% absolute improvement in AUC for arrhythmia classification on PTB-XL and ICBEB datasets compared to prior self-supervised learning methods.
• ESI improves subject identification performance by 3.5% compared to strong self-supervised baselines, demonstrating better generalization to individual ECG patterns.
• Performance increases significantly with larger pretraining datasets, with the most substantial gains observed in early stages of data scaling.
• The Maximum Mean Discrepancy (MMD) between pretraining and test sets decreases as pretraining data size increases, indicating reduced distribution shift and improved generalization.
• Introducing ECG-text misalignment severely degrades model performance, with misalignment rates above 20% leading to performance worse than random initialization, highlighting the critical role of accurate pairing.
• The ablation study confirms that both contrastive and captioning losses are essential, as joint optimization leads to superior representations compared to single-loss training.