[Paper Review] Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

Ankit Pal, Malaikannan Sankarasubbu|arXiv (Cornell University)|Feb 10, 2024

Mental Health via Writing11 citations

TL;DR

The paper evaluates Google Gemini against open-source and commercial medical LLMs across medical reasoning, hallucination risk (Med-HALT), and medical VQA, using MultiMedQA, Med-HALT, and NEJM-VQA benchmarks, and provides prompting strategies and evaluation tooling. Gemini lags behind Med-PaLM 2 and GPT-4 in several medical tasks and shows notable hallucination risks, with improvements from prompting techniques.

ABSTRACT

Large language models have the potential to be valuable in the healthcare industry, but it's crucial to verify their safety and effectiveness through rigorous evaluation. For this purpose, we comprehensively evaluated both open-source LLMs and Google's new multimodal LLM called Gemini across Medical reasoning, hallucination detection, and Medical Visual Question Answering tasks. While Gemini showed competence, it lagged behind state-of-the-art models like MedPaLM 2 and GPT-4 in diagnostic accuracy. Additionally, Gemini achieved an accuracy of 61.45\% on the medical VQA dataset, significantly lower than GPT-4V's score of 88\%. Our analysis revealed that Gemini is highly susceptible to hallucinations, overconfidence, and knowledge gaps, which indicate risks if deployed uncritically. We also performed a detailed analysis by medical subject and test type, providing actionable feedback for developers and clinicians. To mitigate risks, we applied prompting strategies that improved performance. Additionally, we facilitated future research and development by releasing a Python module for medical LLM evaluation and establishing a dedicated leaderboard on Hugging Face for medical domain LLMs. Python module can be found at https://github.com/promptslab/RosettaEval

Motivation & Objective

• Assess Gemini's medical reasoning capabilities across textual and visual modalities.
• Quantify Gemini's hallucination tendencies and safety risks in medical contexts.
• Compare Gemini with open-source LLMs and commercial models on medical benchmarks.
• Provide actionable feedback for prompting strategies and model improvements.
• Release evaluation tooling and a medical-domain leaderboard to promote reproducible research.

Proposed method

• Evaluate Gemini Pro via the Gemini Pro developer API with deterministic settings (temperature 0.0, top-p 1.0).
• Benchmark on MultiMedQA, Med-HALT, and Medical Visual Question Answering using prompting techniques (zero-shot, few-shot, chain-of-thought, self-consistency, ensemble refinement).
• Compare against open-source LLMs (Llama variants, Mistral, Yi-34b, Qwen-72b, etc.) and closed models (MedPaLM, MedPaLM 2, GPT-4).
• Use accuracy as primary metric across MultiMedQA and VQA; use Pointwise Score for Med-HALT evaluation.
• Introduce RosettaEval Python module for medical LLM evaluation and a Hugging Face leaderboard for medical LLMs.

Figure A.1: The MultiMedQA score of the Med-PaLM 2, GPT-4 and Gemini Pro, where the detailed performance of MultiMedQA in Section A.4.1

Experimental results

Research questions

• RQ1 How accurately can Gemini solve complex medical reasoning problems across text and images?
• RQ2 Does Gemini exhibit hallucinations or overconfidence in medical responses?
• RQ3 How does Gemini's performance compare to open-source and commercial medical LLMs on standard benchmarks?
• RQ4 What prompting strategies improve Gemini's medical reasoning and reduce hallucinations?
• RQ5 What domain-specific strengths and weaknesses exist in Gemini's medical knowledge and multimodal capabilities?

Key findings

• Gemini Pro achieves strong but not state-of-the-art performance on MultiMedQA benchmarks compared with Med-PaLM 2 and GPT-4 (e.g., MedQA USMLE and MedMCQA results discussed).
• On Med-HALT, Gemini shows high accuracy for Reasoning Fake but low protection against overconfidence (Reasoning FCT) and memory-based retrieval challenges (IR PubMedlink tasks).
• Medical VQA accuracy for Gemini Pro is 61.45%, significantly below GPT-4V's 88%.
• Subject-wise analysis shows Gemini excels in Biostatistics, Epidemiology, Cell Biology, and certain domains (e.g., Gastroenterology, O&G) but underperforms in other medical subfields and complex reasoning tasks.
• Prompting strategies such as Chain-of-Thought and Ensemble Refinement can improve performance in specific domains, though effects vary by dataset; zero-shot and few-shot prompts yield different gains across tasks.
• The authors provide a Python evaluation module (RosettaEval) and establish a Hugging Face leaderboard to support reproducible medical LLM research.

Figure A.2: Illustration of the ensemble model, known as self-consistency. In this method, the LLM generates multiple responses and selects the most frequent one as the final answer.