[论文解读] Towards General Purpose Medical AI: Continual Learning Medical Foundation Model

Huahui Yi, Ziyuan Qin+6 more|arXiv (Cornell University)|Mar 12, 2023

Domain Adaptation and Few-Shot Learning被引用 11

一句话总结

本文将视觉-语言模型作为医疗基础模型，在跨领域与跨任务的情境下进行评估，结果显示带回放缓冲的持续学习可提升跨领域/跨任务的泛化能力并减缓灾难性遗忘。

ABSTRACT

Inevitable domain and task discrepancies in real-world scenarios can impair the generalization performance of the pre-trained deep models for medical data. Therefore, we audaciously propose that we should build a general-purpose medical AI system that can be seamlessly adapted to downstream domains/tasks. Since the domain/task adaption procedures usually involve additional labeling work for the target data, designing a data-efficient adaption algorithm is desired to save the cost of transferring the learned knowledge. Our recent work found that vision-language models (VLMs) are efficient learners with extraordinary cross-domain ability. Therefore, in this work, we further explore the possibility of leveraging pre-trained VLMs as medical foundation models for building general-purpose medical AI, where we thoroughly investigate three machine-learning paradigms, i.e., domain/task-specialized learning, joint learning, and continual learning, for training the VLMs and evaluate their generalization performance on cross-domain and cross-task test sets. To alleviate the catastrophic forgetting during sequential training, we employ rehearsal learning and receive a sharp boost in terms of generalization capability. In a nutshell, our empirical evidence suggests that continual learning may be a practical and efficient learning paradigm for the medical foundation model. And we hope researchers can use our empirical evidence as basement to further explore the path toward medical foundation model.

研究动机与目标

• 推动一种可适应多样领域和任务且无需大量标注的通用医疗AI系统。
• 研究面向医疗VLM的三种学习范式（领域/任务专用、联合学习、持续学习）。
• 评估VLM在医疗影像任务中的跨领域与跨任务泛化能力。
• 识别在持续学习中减缓灾难性遗忘的机制（回放/复现）。

提出的方法

• 使用经过预训练的视觉-语言模型（VLMs）作为医疗基础模型，并结合文本提示。
• 比较领域/任务专用、联合学习和持续学习范式，在异构医疗数据上训练VLMs。
• 采用带回放缓冲的回放学习，以在持续学习中缓解灾难性遗忘。
• 在跨领域的息肉数据集和跨任务的医疗数据集上评估泛化能力，涵盖多个任务（息肉检测、海马体、甲状腺结节等）。
• 讨论利用LLMs和VLMs进行基于提示的适配，以为医疗概念生成有信息量的提示。

实验结果

研究问题

• RQ1 领域/任务专用的VLMs是否能泛化到未见过的医疗领域/任务？
• RQ2 联合学习是否能提升跨领域/跨任务的泛化能力，以及它的数据需求是什么？
• RQ3 带回放的持续学习是否能缓解遗忘并在医疗基础模型中实现有竞争力的泛化？
• RQ4 将VLMs适配到医疗领域的实际提示与数据策略有哪些？
• RQ5 领域变动和任务变动如何影响基于VLM的医疗基础模型的跨域和跨任务性能？

主要发现

• 领域/任务专用模型在跨领域/跨任务泛化方面表现不佳。
• 联合学习提升泛化，但需要获得多样化的大型异构数据。
• 顺序训练会造成严重的灾难性遗忘，削弱跨领域性能。
• 带回放缓冲的回放学习显著提升跨域/跨任务泛化，通常超过某些联合学习设置。
• 带回放的持续学习为开发能够处理多领域和多任务的医疗基础模型提供了一个实用路径。