[Paper Review] Med-R$^2$: Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine

Keer Lu, Liang Zheng+7 more|ArXiv.org|Jan 21, 2025

Ethics in Clinical Research3 citations

TL;DR

Med-R2 presents an evidence-based medicine (EBM) aligned retrieval and reasoning framework that enhances LLM medical performance using a coarse-to-fine evidence search, reranking, and CoT-guided reasoning without extra training costs.

ABSTRACT

Large Language Models (LLMs) have exhibited remarkable capabilities in clinical scenarios. Despite their potential, existing works face challenges when applying LLMs to medical settings. Strategies relying on training with medical datasets are highly cost-intensive and may suffer from outdated training data. Leveraging external knowledge bases is a suitable alternative, yet it faces obstacles such as limited retrieval precision and poor effectiveness in answer extraction. These issues collectively prevent LLMs from demonstrating the expected level of proficiency in mastering medical expertise. To address these challenges, we introduce Med-R^2, a novel LLM physician framework that adheres to the Evidence-Based Medicine (EBM) process, efficiently integrating retrieval mechanisms as well as the selection and reasoning processes of evidence, thereby enhancing the problem-solving capabilities of LLMs in healthcare scenarios and fostering a trustworthy LLM physician. Our comprehensive experiments indicate that Med-R^2 achieves a 13.27\% improvement over vanilla RAG methods and even a 4.55\% enhancement compared to fine-tuning strategies, without incurring additional training costs. Furthermore, we find that our LLaMA3.1-70B + Med-R$^2$ surpasses frontier models, including GPT-4o, Claude3.5-Sonnet and DeepSeek-V3 by 1.05\%, 6.14\% and 1.91\%. Med-R$^2$ effectively enhances the capabilities of LLMs in the medical domain.

Motivation & Objective

• Highlight challenges in applying LLMs to medicine due to costly domain-specific training, limited retrieval precision, and poor answer extraction.
• Propose Med-R2, an LLM physician framework grounded in Evidence-Based Medicine (EBM).
• Show that retrieval and reasoning integrated with EBM improves medical problem-solving without additional training costs.
• Demonstrate robustness across multiple open-source LLMs and context window scales.

Proposed method

• Build a four-stage Med-R2 pipeline aligned with EBM: question formulation, evidence retrieval and appraisal, evidence application, and effect assessment.
• Construct a medical knowledge base with four resource types (academic papers, entries, books, guidelines) and segment long documents for retrieval.
• Use a hybrid retriever (dense + sparse) and a coarse-to-fine reranker with an evidence hierarchy and usefulness scoring.
• Generate chain-of-thought (CoT) demonstrations from evidence to aid reformulation and few-shot learning for the target model.
• Apply a coarse-to-fine document filtering and a 16-category document-type scoring to match answer structure with query intent.
• Evaluate using multiple open-source LLMs (Qwen, LLaMA, etc.) across datasets (MedQA-USMLE, MedQA-MCMLE, MedMCQA, PubMedQA, MMLU-Med).

Experimental results

Research questions

• RQ1 How does an EBM-guided retrieval and reasoning framework affect medical question answering accuracy across diverse datasets?
• RQ2 What impact does combining dense and sparse retrieval have on medical evidence precision and answer quality?
• RQ3 How do context window size and model scale interact with Med-R2 performance in medical domains?
• RQ4 Can Med-R2 reduce the need for expensive model fine-tuning while maintaining or surpassing fine-tuned baselines?
• RQ5 What is the contribution of CoT integration in retrieval versus post-retrieval reasoning?

Key findings

• Med-R2 achieves average improvements over direct-response baselines and vanilla RAG across models and datasets.
• Across all tested models, Med-R2 yields 14.87% higher performance than vanilla RAG and 3.59% higher than fine-tuning without extra training costs.
• Lightweight models (7B–8B) gain particularly large benefits from external knowledge augmentation (around 78–80% relative gains).
• Med-R2 outperforms fine-tuning on cross-dataset evaluation, highlighting better generalization through external knowledge and retrieval.
• CoT-enhanced variants show context-window and scale-dependent effects, with larger models and longer contexts showing more robust gains; CoT at initial retrieval can underperform for smaller models.