[论文解读] OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine

Xiaosong Wang, Xiaofan Zhang+18 more|arXiv (Cornell University)|Feb 28, 2024

Biomedical Text Mining and Ontologies被引用 8

一句话总结

OpenMEDLab 是一个开源平台，整合跨模态的医疗基础模型（图像、文本、蛋白质），通过提示、预训练、评估和基准测试，能够适应医疗任务。

ABSTRACT

The emerging trend of advancing generalist artificial intelligence, such as GPTv4 and Gemini, has reshaped the landscape of research (academia and industry) in machine learning and many other research areas. However, domain-specific applications of such foundation models (e.g., in medicine) remain untouched or often at their very early stages. It will require an individual set of transfer learning and model adaptation techniques by further expanding and injecting these models with domain knowledge and data. The development of such technologies could be largely accelerated if the bundle of data, algorithms, and pre-trained foundation models were gathered together and open-sourced in an organized manner. In this work, we present OpenMEDLab, an open-source platform for multi-modality foundation models. It encapsulates not only solutions of pioneering attempts in prompting and fine-tuning large language and vision models for frontline clinical and bioinformatic applications but also building domain-specific foundation models with large-scale multi-modal medical data. Importantly, it opens access to a group of pre-trained foundation models for various medical image modalities, clinical text, protein engineering, etc. Inspiring and competitive results are also demonstrated for each collected approach and model in a variety of benchmarks for downstream tasks. We welcome researchers in the field of medical artificial intelligence to continuously contribute cutting-edge methods and models to OpenMEDLab, which can be accessed via https://github.com/openmedlab.

研究动机与目标

• 促进一个统一的平台，跨模态分享医疗基础模型和数据。
• 展示用于医疗下游任务的提示、微调和适配技术。
• 提供大规模医疗数据集和基准测试，以评估泛化性和效率。
• 鼓励社区贡献，推动医学领域的领域特定基础模型的发展。

提出的方法

• 策划并发布覆盖LLM、成像和蛋白质工程的医疗基础模型预训练模型套件。
• 开发提示与适配管线，将通用模型转移到医疗领域（提示、RAG、记忆、受限生成）。
• 汇编多模态数据集和基准，用于医疗影像与NLP的模型评估与适配。
• 使用结构化基准和专用的医疗 LLM 评估框架进行评估（EL0 风格设置与 Elo 评级）。
• 提供自动化、云端评估平台（MedBench）和用于可重复性的开源工具。

Figure 1 : OpenMEDLab: an open-source platform for facilitating medical foundation models.

实验结果

研究问题

• RQ1 如何通过提示和领域特定微调，将通用基础模型有效适配到多样的医疗模态？
• RQ2 哪些基准和数据集最能评估跨模态的医疗基础模型的泛化性、效率和安全性？
• RQ3 开源平台能否以更低成本和更高鲁棒性加速医疗基础模型的开发与部署？

主要发现

• OpenMEDLab 提供涵盖成像、NLP 与蛋白领域的多样化开源医疗基础模型。
• 突出模型包括 LLM（如 PULSE）以及多种视觉/3D/医疗影像模型（如 RETFound、Endo-FM、MIS-FM、STU-Net、SAM-Med3D、BROW、PathoDuet、D-MIM、USFM）。
• 使用 EL0 风格 Elo 基于的评估来比较医疗 LLM，在若干基准上 GPT-4 取得平均排名第一。
• MedBench 提供一个拥有超过 300k 问题的中文医疗 LLM 评估套件，覆盖医疗理解、生成、问答与伦理。
• 有前景的提示策略（CITE、MIU-VL）和小样本定位（MedLAM）展示将基础模型 grounding in medical data 的实际路径。

Figure 2 : The overall organization of OpenMEDLab.