[Paper Review] ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning

Seokmin Choi, Sajad Mousavi+4 more|arXiv (Cornell University)|Jun 10, 2023

ECG Monitoring and AnalysisMedicine3 citations

TL;DR

ECGBERT introduces a self-supervised representation learning framework inspired by BERT to unlock the 'hidden language' of ECG signals using large-scale unlabeled data. By pre-training on raw ECGs and fine-tuning with minimal labeled data, ECGBERT achieves state-of-the-art performance across diverse tasks like atrial fibrillation detection, heartbeat classification, sleep apnea detection, and user authentication.

ABSTRACT

In the medical field, current ECG signal analysis approaches rely on supervised deep neural networks trained for specific tasks that require substantial amounts of labeled data. However, our paper introduces ECGBERT, a self-supervised representation learning approach that unlocks the underlying language of ECGs. By unsupervised pre-training of the model, we mitigate challenges posed by the lack of well-labeled and curated medical data. ECGBERT, inspired by advances in the area of natural language processing and large language models, can be fine-tuned with minimal additional layers for various ECG-based problems. Through four tasks, including Atrial Fibrillation arrhythmia detection, heartbeat classification, sleep apnea detection, and user authentication, we demonstrate ECGBERT's potential to achieve state-of-the-art results on a wide variety of tasks.

Motivation & Objective

• To address the critical challenge of limited labeled ECG data in medical AI by leveraging vast amounts of unlabeled ECG signals.
• To develop a universal, transferable deep learning model for ECG analysis that reduces dependency on task-specific data and manual feature engineering.
• To explore whether ECG signals can be interpreted as a 'language' akin to natural language, enabling application of large language model techniques.
• To demonstrate the versatility and robustness of a single pre-trained model across multiple downstream ECG-based medical tasks.
• To establish a reproducible, non-black-box framework for ECG representation learning that enhances accountability in medical AI.

Proposed method

• ECGBERT adapts the BERT transformer architecture to ECG signals by treating segmented ECG waves as 'tokens' in a sequence, enabling bidirectional context learning.
• The model uses a two-stage training process: first, ECG signals are segmented into R-peaks and QRS complexes using a dedicated wave segmentation module, forming token sequences.
• During pre-training, a masked autoencoder objective is applied—randomly masking 15% of the ECG tokens and training the model to reconstruct them, learning general ECG patterns.
• The pre-trained ECGBERT encoder is then fine-tuned with minimal additional layers (e.g., two dense layers) for specific downstream tasks such as arrhythmia detection or user identification.
• The framework supports multi-lead input in principle, though current experiments use only lead-II ECGs due to data availability constraints.
• A contrastive learning objective is implicitly embedded via the masked reconstruction task, encouraging the model to learn disentangled, semantically meaningful representations of ECG morphology.

Experimental results

Research questions

• RQ1 Can ECG signals be effectively modeled as a sequence of tokens analogous to natural language, enabling application of large language model techniques?
• RQ2 Can self-supervised pre-training on large-scale unlabeled ECG data significantly improve downstream performance with minimal labeled data?
• RQ3 Does a single pre-trained ECGBERT model generalize across diverse ECG-based tasks such as arrhythmia detection, sleep apnea diagnosis, and biometric authentication?
• RQ4 How does ECGBERT’s performance compare to supervised baselines like ResNet across multiple ECG tasks with limited annotations?
• RQ5 What are the key limitations of the current framework, particularly regarding signal quality, multi-lead support, and segmentation robustness?

Key findings

• ECGBERT achieved state-of-the-art accuracy of 0.938 on the user identification task, outperforming the ResNet baseline (0.920) despite using minimal labeled data.
• On atrial fibrillation detection, ECGBERT achieved an F1-score of 0.882, surpassing the ResNet baseline, demonstrating superior generalization on arrhythmia classification.
• For sleep apnea detection, ECGBERT achieved a sensitivity of 0.831, significantly higher than the ResNet baseline (0.653), indicating better detection of positive cases.
• ECGBERT demonstrated stable and consistent performance across four distinct ECG tasks, validating its transferability and robustness as a universal ECG representation model.
• The model’s performance was particularly strong in tasks requiring high sensitivity, such as arrhythmia detection, where missing positive cases is clinically critical.
• Despite challenges from noisy signals and short 10-second sub-sequence inputs in the Apnea dataset, ECGBERT maintained higher overall accuracy (0.725) than ResNet (0.709), indicating resilience to data variability.