[Paper Review] MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations

Congbo Ma, Yichun Zhang+7 more|arXiv (Cornell University)|Feb 5, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

MedErrBench introduces a multilingual benchmark (English, Arabic, Chinese) for medical error detection, localization, and correction with clinician-annotated error types across 10 categories. It evaluates a broad set of LLMs to reveal multilingual gaps and the need for clinically grounded, language-aware models.

ABSTRACT

Inaccuracies in existing or generated clinical text may lead to serious adverse consequences, especially if it is a misdiagnosis or incorrect treatment suggestion. With Large Language Models (LLMs) increasingly being used across diverse healthcare applications, comprehensive evaluation through dedicated benchmarks is crucial. However, such datasets remain scarce, especially across diverse languages and contexts. In this paper, we introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction, developed under the guidance of experienced clinicians. Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese, with natural clinical cases annotated and reviewed by domain experts. We assessed the performance of a range of general-purpose, language-specific, and medical-domain language models across all three tasks. Our results reveal notable performance gaps, particularly in non-English settings, highlighting the need for clinically grounded, language-aware systems. By making MedErrBench and our evaluation protocols publicly-available, we aim to advance multilingual clinical NLP to promote safer and more equitable AI-based healthcare globally. The dataset is available in the supplementary material. An anonymized version of the dataset is available at: https://github.com/congboma/MedErrBench.

Motivation & Objective

• Develop a clinician-informed taxonomy of 10 clinical error types for multilingual medical NLP evaluation.
• Create and validate a multilingual benchmark (English, Arabic, Chinese) for medical error detection, localization, and correction.
• Assess a broad range of general-purpose, language-specialized, and medical-domain LLMs on the benchmark across three languages.
• Provide insights into model limitations, cross-language generalization, and the impact of prompts and few-shot learning on performance.

Proposed method

• Partition multilingual clinical data into English, Chinese, and Arabic using non-translation, multi-source collection.
• Define a 10-type error taxonomy extending MEDEC with five new categories (Lab/Serum Value Interpretation, Physiology, Histology, Anatomy, Epidemiology) and provide definitions and examples.
• Inject clinically grounded errors into notes to create errorful and corrected pairs for detection, localization, and correction tasks.
• Annotate each instance with importance of clinical words, difficulty level (Easy/Medium/Hard), and reasoning type (Factual Recall, Single-hop, Multi-hop).
• Perform two-stage clinician review for content validity and annotation quality; resolve disagreements and ensure accuracy.

Figure 1: Overview of MedErrBench.

Experimental results

Research questions

• RQ1 What is the performance of a broad spectrum of LLMs on multilingual medical error detection, localization, and correction across English, Arabic, and Chinese?
• RQ2 How do error-type definitions, exemplar prompts, and few-shot examples influence model performance in clinical error tasks?
• RQ3 How do knowledge-based versus scenario-based clinical notes affect model capabilities in multilingual settings?
• RQ4 What are cross-lingual generalization capabilities and language-specific challenges for medical error detection and correction?
• RQ5 What are the limitations of current models and what directions improve clinically grounded, language-aware systems?

Key findings

• Doubao-1.5-thinking-pro, Deepseek-R1, and Deepseek-V3 outperform others across languages on several tasks.
• Medical-domain LLMs do not consistently outperform general-purpose models on error detection/correction tasks.
• Arabic performance is notably weaker for some models, highlighting domain adaptation gaps in low-resource language settings.
• Providing error-type definitions and few-shot exemplars generally improves performance, with definitions particularly beneficial in zero-shot settings.
• Localization and correction are more challenging than detection across languages, and prompt design exhibits model-specific effects.
• Human evaluation showed clinician-annotated Chinese samples favor certain models (e.g., Gemini 2.0 Flash) over others (e.g., GPT-4o-mini).

Figure 2: Distribution of difficulty level and reasoning type.