[Paper Review] OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine

Xiaosong Wang, Xiaofan Zhang+18 more|arXiv (Cornell University)|Feb 28, 2024

Biomedical Text Mining and Ontologies8 citations

TL;DR

OpenMEDLab is an open-source platform that bundles medical foundation models across modalities (images, text, protein) with prompting, pre-training, evaluation, and benchmarks, enabling adaptation to medical tasks.

ABSTRACT

The emerging trend of advancing generalist artificial intelligence, such as GPTv4 and Gemini, has reshaped the landscape of research (academia and industry) in machine learning and many other research areas. However, domain-specific applications of such foundation models (e.g., in medicine) remain untouched or often at their very early stages. It will require an individual set of transfer learning and model adaptation techniques by further expanding and injecting these models with domain knowledge and data. The development of such technologies could be largely accelerated if the bundle of data, algorithms, and pre-trained foundation models were gathered together and open-sourced in an organized manner. In this work, we present OpenMEDLab, an open-source platform for multi-modality foundation models. It encapsulates not only solutions of pioneering attempts in prompting and fine-tuning large language and vision models for frontline clinical and bioinformatic applications but also building domain-specific foundation models with large-scale multi-modal medical data. Importantly, it opens access to a group of pre-trained foundation models for various medical image modalities, clinical text, protein engineering, etc. Inspiring and competitive results are also demonstrated for each collected approach and model in a variety of benchmarks for downstream tasks. We welcome researchers in the field of medical artificial intelligence to continuously contribute cutting-edge methods and models to OpenMEDLab, which can be accessed via https://github.com/openmedlab.

Motivation & Objective

• Promote a cohesive platform for sharing medical foundation models and data across modalities.
• Demonstrate prompting, fine-tuning, and adaptation techniques for medical downstream tasks.
• Provide large-scale medical datasets and benchmarking to assess generalization and efficiency.
• Encourage community contributions to advance domain-specific foundation models in medicine.

Proposed method

• Curate and publish a suite of pre-trained medical foundation models spanning LLMs, imaging, and protein engineering.
• Develop prompting and adaptation pipelines to transfer generalist models to medical domains (prompting, RAG, memory, constrained generation).
• Assemble multi-modal datasets and benchmarks for model evaluation and adaptation in medical imaging and NLP.
• Evaluate models using structured benchmarks and a dedicated medical LLM evaluation framework (EL0-like setup and Elo-based ranking).
• Offer automatic, cloud-based evaluation platforms (MedBench) and open-source tooling for reproducibility.

Figure 1 : OpenMEDLab: an open-source platform for facilitating medical foundation models.

Experimental results

Research questions

• RQ1 How can generalist foundation models be effectively adapted to diverse medical modalities via prompting and domain-specific fine-tuning?
• RQ2 What benchmarks and datasets best assess generalization, efficiency, and safety of medical foundation models across modalities?
• RQ3 Can open-source platforms accelerate the development and deployment of medical foundation models with lower costs and higher robustness?

Key findings

• OpenMEDLab provides a diverse set of open-source medical foundation models across imaging, NLP, and protein domains.
• Prominent models include LLMs (e.g., PULSE) and multiple vision/3D/medical image models (e.g., RETFound, Endo-FM, MIS-FM, STU-Net, SAM-Med3D, BROW, PathoDuet, D-MIM, USFM).
• EL0-style Elo-based evaluation is used to compare medical LLMs, with GPT-4 achieving the top average rank on several benchmarks.
• MedBench offers a Chinese medical LLM evaluation suite with over 300k questions, addressing medical understanding, generation, QA, and ethics.
• Promising prompting strategies (CITE, MIU-VL) and few-shot localization (MedLAM) illustrate practical paths to grounding foundation models in medical data.

Figure 2 : The overall organization of OpenMEDLab.