[Paper Review] Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

Shefayat E Shams Adib, Ahmed Alfey Sani+3 more|arXiv (Cornell University)|Feb 16, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper benchmarks five LLMs on medical QA using zero-shot evaluation with iCliniq data, comparing automatic metrics (BLEU/ROUGE) and LLM-as-a-Judge assessments to gauge medical accuracy and safety. Larger models show better performance, with Llama 3.3 70B Instruct leading.

ABSTRACT

Recently, Large Language Models (LLMs) have gained significant traction in medical domain, especially in developing a QA systems to Medical QA systems for enhancing access to healthcare in low-resourced settings. This paper compares five LLMs deployed between April 2024 and August 2025 for medical QA, using the iCliniq dataset, containing 38,000 medical questions and answers of diverse specialties. Our models include Llama-3-8B-Instruct, Llama 3.2 3B, Llama 3.3 70B Instruct, Llama-4-Maverick-17B-128E-Instruct, and GPT-5-mini. We are using a zero-shot evaluation methodology and using BLEU and ROUGE metrics to evaluate performance without specialized fine-tuning. Our results show that larger models like Llama 3.3 70B Instruct outperform smaller models, consistent with observed scaling benefits in clinical tasks. It is notable that, Llama-4-Maverick-17B exhibited more competitive results, thus highlighting evasion efficiency trade-offs relevant for practical deployment. These findings align with advancements in LLM capabilities toward professional-level medical reasoning and reflect the increasing feasibility of LLM-supported QA systems in the real clinical environments. This benchmark aims to serve as a standardized setting for future study to minimize model size, computational resources and to maximize clinical utility in medical NLP applications.

Motivation & Objective

• Provide a comprehensive zero-shot benchmark of modern LLMs for medical QA on a large real-world dataset (iCliniq).
• Assess correlations between model size/architecture and medical QA performance.
• Introduce and validate a standardized dual-evaluation framework combining automatic metrics with LLM-as-a-Judge clinical quality assessment.
• Offer deployment guidance balancing accuracy and resource constraints for clinical settings.

Proposed method

• Use a zero-shot evaluation protocol with a standardized medical prompt across five LLMs.
• Evaluate on a 3,000-question subset of the 38,000 iCliniq Medical QA dataset.
• Compute BLEU and ROUGE metrics to assess lexical similarity and coverage.
• Apply an LLM-as-a-Judge framework (Claude Sonnet 4) to rate Medical Accuracy, Completeness, Safety, Clarity, and Helpfulness using a 5-point scale with weighted scoring (30/25/20/15/10).
• Compare results against MedLM baselines from prior work to contextualize improvements.

Experimental results

Research questions

• RQ1 How do five contemporary LLMs perform on zero-shot medical QA tasks using the iCliniq dataset?
• RQ2 What is the relationship between model size/architecture and medical QA performance in zero-shot settings?
• RQ3 How does LLM-as-a-Judge evaluation align with traditional BLEU/ROUGE metrics in medical QA?
• RQ4 What deployment guidance can be derived for high-accuracy clinical environments vs. resource-constrained settings?

Key findings

• Llama 3.3 70B Instruct achieves the highest BLEU-1, ROUGE-1, and ROUGE-L among the models evaluated.
• Llama-4-Maverick 17B shows competitive efficiency, performing close to the 70B model with far fewer parameters.
• GPT-5-mini performs poorly across automatic metrics, highlighting possible implementation/configuration issues.
• There is a clear positive correlation between model size and medical QA performance, with architectural innovations enabling smaller models to approach larger ones.
• LLM-as-a-Judge results align with automatic metrics, reinforcing the ranking and validating the evaluation framework.
• Medical accuracy is highest for the top model (4.83/5), while safety is highest for GPT-5-mini (3.80/5) despite weak lexical metrics.