[论文解读] EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation

Miloš Vukadinovic, Xiu Tang+6 more|arXiv (Cornell University)|Oct 13, 2024

Lung Cancer Diagnosis and Treatment被引用 13

一句话总结

EchoPrime 是一个多视图、基于视频的视觉–语言模型，训练于超过 12 million 视频-报告对，以在标准视图和疾病上执行全方位的经胸超声心动图解读，达到 23 个基准的最先进结果。

ABSTRACT

Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. Artificial intelligence (AI) in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.

研究动机与目标

• Motivate automated, comprehensive echocardiography interpretation that leverages multiple views from a full exam.
• Develop a unified vision-language embedding model that handles all standard echocardiographic views.
• Enable view-informed attention and retrieval-augmented interpretation to synthesize information across videos.
• Evaluate performance across diverse datasets and disease representations to demonstrate generalization.
• Compare against task-specific and prior foundation models to establish state-of-the-art results.

提出的方法

• Train a unified embedding model using contrastive learning across all standard echocardiographic views.
• Incorporate a view-classification module to identify the echocardiographic view for each video.
• Implement a view-informed anatomic attention mechanism to weight video-specific interpretations by view.
• Use retrieval-augmented interpretation to integrate information from all videos in a comprehensive study.
• Evaluate on 23 benchmarks of cardiac form and function using datasets from two independent healthcare systems.

实验结果

研究问题

• RQ1 Can a single vision-language model effectively interpret comprehensive echocardiography by leveraging multiple views from a full study?
• RQ2 Does view-informed attention improve mapping between echocardiographic views, anatomical structures, and clinical interpretations?
• RQ3 How does retrieval-augmented interpretation perform when synthesizing information across all videos in a study?
• RQ4 What is the comparative performance of EchoPrime against task-specific models and prior foundation models on diverse cardiac benchmarks?

主要发现

• Achieves state-of-the-art performance on 23 diverse benchmarks of cardiac form and function.
• Surpasses both task-specific approaches and prior foundation models in evaluations across two independent health-system datasets.
• Demonstrates effective comprehensive echocardiography interpretation through multi-view, view-informed, video-based capabilities.