[论文解读] RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis

Xiaoman Zhang, Chaoyi Wu+5 more|arXiv (Cornell University)|Apr 25, 2024

Lung Cancer Diagnosis and Treatment被引用 5

一句话总结

RadGenome-Chest CT 是一个基于 CT-RATE 构建的大型、区域定位的胸部 CT 数据集，提供 197 个器官级分割掩模、665k 条多粒度的定位报告，以及 1.3M 条定位的 VQA 对，旨在实现基于区域的文本生成和多模态医学基础模型。

ABSTRACT

Developing generalist foundation model has recently attracted tremendous attention among researchers in the field of AI for Medicine (AI4Medicine). A pivotal insight in developing these models is their reliance on dataset scaling, which emphasizes the requirements on developing open-source medical image datasets that incorporate diverse supervision signals across various imaging modalities. In this paper, we introduce RadGenome-Chest CT, a comprehensive, large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. Specifically, we leverage the latest powerful universal segmentation and large language models, to extend the original datasets (over 25,692 non-contrast 3D chest CT volume and reports from 20,000 patients) from the following aspects: (i) organ-level segmentation masks covering 197 categories, which provide intermediate reasoning visual clues for interpretation; (ii) 665 K multi-granularity grounded reports, where each sentence of the report is linked to the corresponding anatomical region of CT volume in the form of a segmentation mask; (iii) 1.3 M grounded VQA pairs, where questions and answers are all linked with reference segmentation masks, enabling models to associate visual evidence with textual explanations. All grounded reports and VQA pairs in the validation set have gone through manual verification to ensure dataset quality. We believe that RadGenome-Chest CT can significantly advance the development of multimodal medical foundation models, by training to generate texts based on given segmentation regions, which is unattainable with previous relevant datasets. We will release all segmentation masks, grounded reports, and VQA pairs to facilitate further research and development in this field.

研究动机与目标

• 通过对胸部 CT 分析使用区域级监督扩展公开数据集，激发通用医疗 AI 的发展。
• 创建一个区域引导的数据集，以实现对 CT 影像的定位报告生成和定位 VQA。
• 提供资源（分割掩模、定位报告、VQA 对），以促成放射学中可解释的多模态模型。

提出的方法

• 使用 SAT 对器官分割（197 个区域）对 3D 胸部 CT 区域进行定位，将 CT-RATE 扩展。
• 通过 GPT-4 与内部的 NER/QA 流水线将放射报告解析为解剖学匹配的句子；将句子与分割掩模关联。
• 通过将发现和印象转换成与分割区域绑定的问答模板来生成区域定位的 VQA 数据。
• 通过对定位报告和 VQA 组件进行人工验证来验证定位质量。

实验结果

研究问题

• RQ1 如何在大规模胸部CT数据集中加入区域级定位，以支持定位的多模态任务？
• RQ2 CT-RATE 能达到的分割规模与质量、区域定位报告及 VQA 对的规模和质量是多少？
• RQ3 区域到文本的关联是否能提升放射学中可解释的多模态医学基础模型？

主要发现

• 数据集包含来自 20,000 名患者的 25,692 位非增强 3D 胸部 CT 体积及报告。
• 为胸部 CT 区域生成了 197 个器官级分割掩模。
• 生成了 665K 条将每个句子链接到分割区域的多粒度定位报告。
• 创建了 1.3M 条定位 VQA 对（区域级和病例级），并在验证集进行了人工验证。