Skip to main content
QUICK REVIEW

[论文解读] Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset

Louis Blankemeier, Ashwin Kumar|arXiv (Cornell University)|Jun 10, 2024
Medical Image Segmentation TechniquesComputer Science被引用 9
一句话总结

Merlin 是一个在成对CT影像、EHR编码和放射报告上训练的3D 视觉-语言基础模型,能够执行零-shot 和适应性任务,包括3D分割和放射报告生成,使用单GPU计算策略。

ABSTRACT

The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings for abdominal CT interpretation, we introduce Merlin, a 3D VLM that learns from volumetric CT scans, electronic health record data and radiology reports. This approach is enabled by a multistage pretraining framework that does not require additional manual annotations. We trained Merlin using a high-quality clinical dataset of paired CT scans (>6 million images from 15,331 CT scans), diagnosis codes (>1.8 million codes) and radiology reports (>6 million tokens). We comprehensively evaluated Merlin on 6 task types and 752 individual tasks that covered diagnostic, prognostic and quality-related tasks. The non-adapted (off-the-shelf) tasks included zero-shot classification of findings (30 findings), phenotype classification (692 phenotypes) and zero-shot cross-modal retrieval (image-to-findings and image-to-impression). The model-adapted tasks included 5-year chronic disease prediction (6 diseases), radiology report generation and 3D semantic segmentation (20 organs). We validated Merlin at scale, with internal testing on 5,137 CT scans and external testing on 44,098 CT scans from 3 independent sites and 2 public datasets. The results demonstrated high generalization across institutions and anatomies. Merlin outperformed 2D VLMs, CT foundation models and off-the-shelf radiology models. We also release our trained models, code, and dataset, available at: https://github.com/StanfordMIMI/Merlin.

研究动机与目标

  • 提高并减轻放射科医生工作负担,通过开发用于腹部CT解读的3D视觉-语言模型。
  • 利用结构化的EHR数据和非结构化的放射报告来监督一个基于CT的VLM,无需额外人工注释。
  • 在广泛的任务集合上评估Merlin,以展示零-shot和适应能力并分析数据需求。
  • 表明Merlin可以在单个GPU上训练,并对外部数据集和公开CT数据集具有泛化能力。

提出的方法

  • 使用成对的CT扫描(六百万张图像以上,来自15,331个CT)、EHR诊断编码(180万以上编码)和放射报告(六百万个令牌)来训练Merlin。
  • 使用诊断编码进行二元交叉熵损失,使用放射报告进行InfoNCE损失,分阶段或多任务方式训练。
  • 使用I3D 基于初始化和3D骨干网对整个3D CT体积端到端处理。
  • 评估零-shot发现分类、表型分类,以及零-shot跨模态检索(未适应化),以及适应性任务包括5年疾病预测、放射报告生成和3D语义分割。
  • 进行消融实验以评估骨干选择、数据类型(EHR与报告)、报告拆分以及训练策略(分阶段 vs 多任务)。
  • 推导数据扩展定律,将预训练数据规模与性能相关联,并展示在单GPU上的训练。
Figure 1 : Overview of Merlin training and evaluation. (a) Merlin training strategy. Diagnosis codes from the EHR are used as labels for Merlin training, with a binary cross entropy loss. Radiology reports are also used for training, with an InfoNCE loss Oord et al. ( 2018 ) . Training with diagnosi
Figure 1 : Overview of Merlin training and evaluation. (a) Merlin training strategy. Diagnosis codes from the EHR are used as labels for Merlin training, with a binary cross entropy loss. Radiology reports are also used for training, with an InfoNCE loss Oord et al. ( 2018 ) . Training with diagnosi

实验结果

研究问题

  • RQ1一个用结构化的EHR数据和非结构化放射报告共同训练的3D视觉-语言模型,是否能在腹部CT解读上超越任务特定基线?
  • RQ2数据类型(EHR编码 vs 放射报告)、模型结构和训练策略如何影响3D医疗VLM的零-shot和适应任务性能?
  • RQ3Merlin达到目标下游性能需要怎样的数据扩展规模,以及单GPU训练如何影响实用性和泛化?

主要发现

  • Merlin在31个发现项的内部平均F1为0.741,外部平均0.647,优于OpenCLIP和BioMedCLIP(p < 0.001)。
  • Merlin在692个表型上达到宏观AUROC 0.812,对258个表型超过0.85 AUROC,对102个表型超过0.9 AUROC。
  • 零-shot跨模态检索(图像到发现和图像到印象)在内部数据上超越基线,并且在外部数据上的泛化比外部基线高5-7倍。
  • 对5年多疾病预测,覆盖6种疾病,在只有100%下游标签时AUROC为0.757,使用10%标签时为0.708,优于在ImageNet预训练的模型。
  • Merlin在3D语义分割方面有提升,尤其是对较小或复杂的器官,并在数据稀缺情形(10%数据)中展示鲁棒性。
  • 使用Merlin的放射报告生成在多项指标上超越RadFM,生成与解剖章节对齐的结构化报告。
Figure 2 : Zero-shot findings classification. (a) Depicts how zero-shot classification is performed where text embeddings from disease presence prompts and disease absence prompts are compare to the image embedding. (b) We compare the performance of OpenCLIP Cherti et al. ( 2022 ) , BioMedCLIP Zhang
Figure 2 : Zero-shot findings classification. (a) Depicts how zero-shot classification is performed where text embeddings from disease presence prompts and disease absence prompts are compare to the image embedding. (b) We compare the performance of OpenCLIP Cherti et al. ( 2022 ) , BioMedCLIP Zhang

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。