[Paper Review] Contrastive Learning for Multi Label ECG Classification with Jaccard Score Based Sigmoid Loss

Junichiro Takahashi, Masataka Sato+2 more|arXiv (Cornell University)|Feb 11, 2026

ECG Monitoring and Analysis0 citations

TL;DR

The paper trains an ECG encoder with SigLIP using real-world data and introduces a Jaccard-based sigmoid loss to enable true multi-label ECG classification, achieving improved micro-F1 and Jaccard scores, with further gains from embedding size and data augmentation.

ABSTRACT

Recent advances in large language models (LLMs) have enabled the development of multimodal medical AI. While models such as MedGemini achieve high accuracy on VQA tasks like USMLE MM, their performance on ECG based tasks remains limited, and some models, such as MedGemma, do not support ECG data at all. Interpreting ECGs is inherently challenging, and diagnostic accuracy can vary depending on the interpreter's experience. Although echocardiography provides rich diagnostic information, it requires specialized equipment and personnel, limiting its availability. In this study, we focus on constructing a robust ECG encoder for multimodal pretraining using real world hospital data. We employ SigLIP, a CLIP based model with a sigmoid based loss function enabling multi label prediction, and introduce a modified loss function tailored to the multi label nature of ECG data. Experiments demonstrate that incorporating medical knowledge in the language model and applying the modified loss significantly improve multi label ECG classification. To further enhance performance, we increase the embedding dimensionality and apply random cropping to mitigate data drift. Finally, per label analysis reveals which ECG findings are easier or harder to predict. Our study provides a foundational framework for developing medical models that utilize ECG data.

Motivation & Objective

• Motivate robust multimodal ECG encoding using real-world hospital data for multi-label prediction.
• Develop and test a modified sigmoid loss that incorporates Jaccard similarity for ECG findings.
• Assess the impact of medical-knowledge-enabled language models on ECG pretraining.
• Explore enhancements such as higher embedding dimensionality and random cropping to mitigate data drift.

Proposed method

• Use a 1D ResNet-18 as the ECG encoder and Qwen3-8B as the language model.
• Replace CrossEntropy-based CLIP loss with a sigmoid-based loss (SigLIP) and introduce a Jaccard-based eye matrix to handle multi-label prediction.
• Train on 33,732 real-world ECGs with 12-lead, 500 Hz, 10 s recordings; text captions describe findings.
• Compare standard SigLIP loss vs. Modified Sigmoid loss incorporating Jaccard similarity in batch interactions.
• Evaluate with Hamming Loss, Precision (Micro), Recall (Micro), F1 Score (Micro), and Jaccard Index; perform ablations on embedding size and data augmentation (random cropping).
• Experiment with language models having varying ECG knowledge to assess domain knowledge impact.

Experimental results

Research questions

• RQ1 Can a SigLIP-based ECG encoder handle multi-label findings from real-world ECG data?
• RQ2 Does a Jaccard-based sigmoid loss improve multi-label ECG classification performance over the standard SigLIP loss?
• RQ3 How do embedding dimensionality and random cropping affect multi-label ECG performance?
• RQ4 What is the impact of ECG-domain knowledge in the language model on pretraining effectiveness?

Key findings

• Modified Sigmoid loss with Jaccard similarity improves multi-label metrics over the standard SigLIP loss (e.g., lower Hamming Loss and higher F1 Micro, higher Jaccard).
• Embedding dimensionality increase to 256 and applying random cropping significantly boosts performance (F1 Micro up to 0.5028 and Jaccard up to 0.3495).
• Baseline SigLIP with 256-d embeddings and random crop (600 epochs, 20k warmup) yields substantial gains over the baseline (F1 Micro 0.503 vs 0.3082).
• Medical knowledge in the language model (Qwen3-8B vs Gemma3-4B) correlates with higher micro-precision/recall and Jaccard with the proposed loss.
• Final model achieves per-label insights, e.g., lowEF and Atrial fibrillation are more predictable, while certain MI-related labels are harder, and performance remains relatively stable when tested on a different hospital’s data.