[Paper Review] Semi-Supervised Multimodal Multi-Instance Learning for Aortic Stenosis Diagnosis

Zhe Huang, Xiaowei Yu+2 more|arXiv (Cornell University)|Mar 9, 2024

Cardiac Valve Diseases and Treatments4 citations

TL;DR

This paper proposes SMMIL, a semi-supervised multimodal multi-instance learning framework that fuses spectral Doppler and 2D cine-loop echocardiographic images to improve aortic stenosis (AS) diagnosis. By leveraging both labeled and abundant unlabeled data across two modalities, SMMIL achieves state-of-the-art performance, attaining 83.5% balanced accuracy on the TMED-2 dataset for 3-level AS classification and outperforming prior methods by over 10 points on key benchmarks.

ABSTRACT

Automated interpretation of ultrasound imaging of the heart (echocardiograms) could improve the detection and treatment of aortic stenosis (AS), a deadly heart disease. However, existing deep learning pipelines for assessing AS from echocardiograms have two key limitations. First, most methods rely on limited 2D cineloops, thereby ignoring widely available Doppler imaging that contains important complementary information about pressure gradients and blood flow abnormalities associated with AS. Second, obtaining labeled data is difficult. There are often far more unlabeled echocardiogram recordings available, but these remain underutilized by existing methods. To overcome these limitations, we introduce Semi-supervised Multimodal Multiple-Instance Learning (SMMIL), a new deep learning framework for automatic interpretation for structural heart diseases like AS. When deployed, SMMIL can combine information from two input modalities, spectral Dopplers and 2D cineloops, to produce a study-level AS diagnosis. During training, SMMIL can combine a smaller labeled set and an abundant unlabeled set of both modalities to improve its classifier. Experiments demonstrate that SMMIL outperforms recent alternatives at 3-level AS severity classification as well as several clinically relevant AS detection tasks.

Motivation & Objective

• To address the challenge of limited labeled echocardiogram data in automated aortic stenosis diagnosis.
• To integrate spectral Doppler and 2D cine-loop imaging modalities for comprehensive study-level diagnosis, mirroring clinical practice.
• To develop a semi-supervised learning framework that leverages 5,386 unlabeled scans to improve model generalization.
• To overcome the limitations of existing models that treat individual images independently rather than synthesizing a holistic diagnosis.
• To evaluate whether off-the-shelf medical foundation models can perform well on AS diagnosis without fine-tuning.

Proposed method

• SMMIL employs a multiple-instance learning (MIL) framework where each echocardiographic study is treated as a bag of image instances, with the goal of predicting a single study-level label.
• The model uses a multimodal attention pooling mechanism to fuse representations from 2D cine-loops and spectral Doppler images at the instance level before aggregating to a bag-level representation.
• A semi-supervised learning strategy is applied using pseudo-labeling and consistency regularization on an unlabeled set of 5,386 scans to improve classifier robustness.
• The framework is trained end-to-end using a contrastive learning objective to align features across modalities and improve representation quality.
• Instance-level features are extracted using a vision transformer backbone, followed by attention-based pooling to emphasize diagnostically relevant frames.
• The final classifier is trained on both labeled data and pseudo-labeled unlabeled data, with uncertainty-aware loss weighting to stabilize training.

Figure 1: Overview of SMMIL . Top: Illustration of the SSL training workflow. In the first iteration, the model is trained on the labeled set. In all subsequent iterations, the model is trained on the union of the labeled set and the selected unlabeled subset by model from the previous iteration. Th

Experimental results

Research questions

• RQ1 Can a multimodal multi-instance learning framework that fuses 2D cine-loops and spectral Doppler improve aortic stenosis diagnosis beyond single-modality models?
• RQ2 To what extent does incorporating unlabeled echocardiogram data via semi-supervised learning enhance diagnostic performance in low-resource medical imaging settings?
• RQ3 How does the proposed SMMIL framework compare to state-of-the-art models in 3-level AS severity classification and binary AS detection tasks?
• RQ4 Can off-the-shelf medical foundation models perform well on echocardiogram-based AS diagnosis without fine-tuning?
• RQ5 Does the inclusion of Doppler imaging significantly improve diagnostic accuracy compared to 2D cine-loops alone?

Key findings

• SMMIL-ID achieved 82.0% balanced accuracy on the All 2D version of TMED-2, outperforming the previous SOTA (SAMIL) by 9.4 percentage points.
• SMMIL-VD achieved 83.5% balanced accuracy on the ViewLOnly 2D version, surpassing the best prior method by over 10 points.
• The addition of spectral Doppler imaging alone improved average balanced accuracy by 9 percentage points compared to 2D-only models.
• Incorporating unlabeled data via semi-supervised learning contributed nearly 4 percentage points in accuracy gain over a supervised-only baseline.
• SMMIL-VD achieved AUROC of 0.98 on the no vs. some AS detection task, outperforming all baselines, including ABMIL and DSMIL.
• Off-the-shelf medical foundation models (Med-Flamingo and Rad-FM) achieved less than 57% balanced accuracy, indicating they are not yet ready for direct deployment in echocardiogram diagnosis.