[Paper Review] EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation

Miloš Vukadinovic, Xiu Tang+6 more|arXiv (Cornell University)|Oct 13, 2024

Lung Cancer Diagnosis and Treatment13 citations

TL;DR

EchoPrime is a multi-view, video-based vision-language model trained on over 12 million video-report pairs to perform holistic echocardiography interpretation across standard views and diseases, achieving state-of-the-art results on 23 benchmarks.

ABSTRACT

Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. Artificial intelligence (AI) in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.

Motivation & Objective

• Motivate automated, comprehensive echocardiography interpretation that leverages multiple views from a full exam.
• Develop a unified vision-language embedding model that handles all standard echocardiographic views.
• Enable view-informed attention and retrieval-augmented interpretation to synthesize information across videos.
• Evaluate performance across diverse datasets and disease representations to demonstrate generalization.
• Compare against task-specific and prior foundation models to establish state-of-the-art results.

Proposed method

• Train a unified embedding model using contrastive learning across all standard echocardiographic views.
• Incorporate a view-classification module to identify the echocardiographic view for each video.
• Implement a view-informed anatomic attention mechanism to weight video-specific interpretations by view.
• Use retrieval-augmented interpretation to integrate information from all videos in a comprehensive study.
• Evaluate on 23 benchmarks of cardiac form and function using datasets from two independent healthcare systems.

Experimental results

Research questions

• RQ1 Can a single vision-language model effectively interpret comprehensive echocardiography by leveraging multiple views from a full study?
• RQ2 Does view-informed attention improve mapping between echocardiographic views, anatomical structures, and clinical interpretations?
• RQ3 How does retrieval-augmented interpretation perform when synthesizing information across all videos in a study?
• RQ4 What is the comparative performance of EchoPrime against task-specific models and prior foundation models on diverse cardiac benchmarks?

Key findings

• Achieves state-of-the-art performance on 23 diverse benchmarks of cardiac form and function.
• Surpasses both task-specific approaches and prior foundation models in evaluations across two independent health-system datasets.
• Demonstrates effective comprehensive echocardiography interpretation through multi-view, view-informed, video-based capabilities.