[논문 리뷰] MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports

Kevin Wu, Eric Q. Wu외 8명|ArXiv.org|2025. 05. 16.

Biomedical Text Mining and Ontologies인용 수 4

한 줄 요약

A open-access dataset and benchmark that evaluate how well LLMs’ diagnostic reasoning aligns with clinician-authored reasoning, and show that fine-tuning on reasoning traces improves both accuracy and reasoning recall.

ABSTRACT

Doctors and patients alike increasingly use Large Language Models (LLMs) to diagnose clinical cases. However, unlike domains such as math or coding, where correctness can be objectively defined by the final answer, medical diagnosis requires both the outcome and the reasoning process to be accurate. Currently, widely used medical benchmarks like MedQA and MMLU assess only accuracy in the final answer, overlooking the quality and faithfulness of the clinical reasoning process. To address this limitation, we introduce MedCaseReasoning, the first open-access dataset for evaluating LLMs on their ability to align with clinician-authored diagnostic reasoning. The dataset includes 14,489 diagnostic question-and-answer cases, each paired with detailed reasoning statements derived from open-access medical case reports. We evaluate state-of-the-art reasoning LLMs on MedCaseReasoning and find significant shortcomings in their diagnoses and reasoning: for instance, the top-performing open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy and mentions only 64% of the clinician reasoning statements (recall). However, we demonstrate that fine-tuning LLMs on the reasoning traces derived from MedCaseReasoning significantly improves diagnostic accuracy and clinical reasoning recall by an average relative gain of 29% and 41%, respectively. The open-source dataset, code, and models are available at https://github.com/kevinwu23/Stanford-MedCaseReasoning.

연구 동기 및 목표

• Provide an open benchmark to evaluate alignment between LLM diagnostic reasoning and clinician-authored reasoning.
• Construct a large, high-quality dataset of real diagnostic cases with clinician-provided reasoning from PubMed Central case reports.
• Assess current state-of-the-art and open-source LLMs on diagnostic accuracy and reasoning recall using the MedCaseReasoning benchmark.
• Demonstrate that supervised fine-tuning on MedCaseReasoning reasoning traces improves both diagnostic accuracy and reasoning recall, and generalizes to NEJM CPC cases.

제안 방법

• Assemble a 14,489-case MedCaseReasoning dataset from free, clinician-authored case reports with differential diagnoses and final diagnoses.
• Convert case reports to QA format and apply multi-step filtering and clinician validation to ensure quality and faithfulness.
• Define Reasoning Recall to quantify overlap between clinician-provided reasons and model reasoning traces.
• Evaluate models with 10-shot prompting using an LLM-as-a-judge (gpt-4o-mini) for diagnostic accuracy.
• Compute reasoning recall by comparing model reasoning traces to ground-truth clinician reasons.
• Fine-tune open-source models (Qwen-2.5-7B-Instruct, LLaMA-3.1-8B-Instruct, MedReason-8B) on stitched reasoning traces for 3 epochs (SFT).
• Compare performance on MedCaseReasoning test set and on NEJM CPC held-out cases to assess generalization.

실험 결과

연구 질문

• RQ1 How well do current reasoning-capable LLMs diagnose real clinical cases and how faithfully do they reproduce clinician-provided reasoning?
• RQ2 Does fine-tuning LLMs on reasoning traces from MedCaseReasoning improve diagnostic accuracy and reasoning recall?
• RQ3 How does performance on MedCaseReasoning correlate with performance on NEJM CPC diagnostic cases?
• RQ4 What is the impact of supervised fine-tuning on open-source medical LLMs’ ability to recall clinical reasoning?
• RQ5 What is the validity of the Reasoning Recall metric as a proxy for diagnostic competence?

주요 결과

• Top models show limited diagnostic reasoning; OpenAI o3 reaches 64.5% 10-shot accuracy on MedCaseReasoning (65% reported in text), while DeepSeek R1 achieves 48.0%.
• Reasoning recall for DeepSeek R1 is about 64.2% on MedCaseReasoning test set; many models recall only a portion of clinician reasoning.
• Fine-tuning on MedCaseReasoning traces yields significant gains: MedReason-8B (SFT) reaches 50.1% 10-shot accuracy, up from 38.2% baseline; Qwen-2.5-7B (SFT) reaches 42.5% 10-shot accuracy, up from 28.5% baseline.
• Fine-tuning also improves NEJM CPC performance, indicating generalization beyond the MedCaseReasoning dataset.
• There is a strong correlation between MedCaseReasoning and NEJM CPC diagnostic performance (Figure 2).
• Reasoning Recall correlates with overall model performance (Pearson r = 0.710, p = 0.0485) and with reasoning trace length (r = 0.790, p = 0.0196).