[Paper Review] Capabilities of Gemini Models in Medicine

Khaled Saab, Tao Tu+65 more|arXiv (Cornell University)|Apr 29, 2024

Simulation Techniques and Applications89 citations

TL;DR

Med-Gemini, a family of medicine-specialized multimodal models built on Gemini, achieves SoTA on 10 of 14 medical benchmarks, surpassing GPT-4 where compared, and demonstrates strong long-context and multimodal capabilities with web search integration and custom encoders.

ABSTRACT

Excellence in a wide variety of medical applications poses considerable challenges for AI, requiring advanced reasoning, access to up-to-date medical knowledge and understanding of complex multimodal data. Gemini models, with strong general capabilities in multimodal and long-context reasoning, offer exciting possibilities in medicine. Building on these core strengths of Gemini, we introduce Med-Gemini, a family of highly capable multimodal models that are specialized in medicine with the ability to seamlessly use web search, and that can be efficiently tailored to novel modalities using custom encoders. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every benchmark where a direct comparison is viable, often by a wide margin. On the popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU (health & medicine), Med-Gemini improves over GPT-4V by an average relative margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context capabilities through SoTA performance on a needle-in-a-haystack retrieval task from long de-identified health records and medical video question answering, surpassing prior bespoke methods using only in-context learning. Finally, Med-Gemini's performance suggests real-world utility by surpassing human experts on tasks such as medical text summarization, alongside demonstrations of promising potential for multimodal medical dialogue, medical research and education. Taken together, our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment in this safety-critical domain.

Motivation & Objective

• Advance clinical reasoning and multimodal understanding for medical AI using Gemini foundations.
• Evaluate Med-Gemini across text, multimodal, and long-context benchmarks to establish SoTA performance.
• Demonstrate the benefits of web search grounding and modality-specific encoders in medical tasks.
• Assess real-world utility in medical note summarization and referral letter generation.
• Highlight safety considerations and need for rigorous validation before deployment.

Proposed method

• Fine-tune Gemini 1.0 Ultra and 1.0 Pro to create Med-Gemini L 1.0 (web-search grounded) and Med-Gemini M 1.0/1.5 for multimodal tasks.
• Develop self-training data with reasoning explanations (CoTs) and chain-of-thought prompts; create MedQA-R and MedQA-RS datasets.
• Implement uncertainty-guided search at inference to trigger web search when needed and incorporate retrieved results into prompts.
• Fine-tune on eight multimodal medical datasets to form Med-Gemini-M 1.5 and create specialized encoders (e.g., ECG) for novel modalities.
• Leverage long-context configurations and chain-of-reasoning to address needle-in-a-haystack EHR retrieval and medical video understanding tasks.
• Benchmark across 25 tasks spanning 14 medical benchmarks, including MedQA (USMLE), NEJM CPC, and GeneTuring.

Figure 1 : Overview of our contributions. We introduce Med-Gemini , a family of highly capable, multimodal medical models built upon Gemini. We enhance our models’ clinical reasoning capabilities through self-training and web search integration, while improving multimodal performance via fine-tuning

Experimental results

Research questions

• RQ1 Can Med-Gemini achieve state-of-the-art results across a wide range of medical tasks (text, multimodal, long-context)?
• RQ2 Does web search grounding and uncertainty-guided inference improve medical reasoning accuracy?
• RQ3 How well can modality-specific encoders extend Med-Gemini to novel medical data (e.g., ECG) and long-context EHRs?
• RQ4 What is the real-world utility of Med-Gemini in tasks like medical note summarization and referral-letter generation?
• RQ5 How does Med-Gemini compare to GPT-4 on viable benchmarks where direct comparisons exist?

Key findings

• Med-Gemini achieves SoTA performance on 10 of 14 medical benchmarks and surpasses GPT-4 where comparisons are available.
• On MedQA (USMLE), the best Med-Gemini model reaches 91.1% accuracy, outperforming Med-PaLM 2 by 4.6%.
• Uncertainty-guided search generalizes to NEJM CPC and GeneTuring benchmarks, achieving SoTA results.
• In multimodal benchmarks, Med-Gemini attains SoTA on 5 of 7 tasks, and improves over GPT-4V on average by 44.5% relative margin.
• Long-context capabilities enable needle-in-a-haystack EHR understanding and medical video QA SoTA results.
• Beyond benchmarks, Med-Gemini shows potential real-world utility in medical note summarization and referral letter generation, with qualitative multimodal dialogue demonstrations.

Figure 2 : Self-training and search tool-use. The left panel illustrates the self-training with search framework used to fine-tune Med-Gemini-L 1.0 for advanced medical reasoning and use of web search. This framework iteratively generates reasoning responses (CoTs) with and without web search, impro