[Paper Review] Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries

Yiqiao Jin, Mohit Chandra+4 more|arXiv (Cornell University)|Oct 19, 2023

Artificial Intelligence in Healthcare and Education10 citations

TL;DR

The paper introduces XLingEval, a cross-lingual framework, and XLingHealth, a multilingual healthcare benchmark to evaluate LLMs across English, Spanish, Chinese, and Hindi, revealing significant language disparities in correctness, consistency, and verifiability.

ABSTRACT

Large language models (LLMs) are transforming the ways the general public accesses and consumes information. Their influence is particularly pronounced in pivotal sectors like healthcare, where lay individuals are increasingly appropriating LLMs as conversational agents for everyday queries. While LLMs demonstrate impressive language understanding and generation proficiencies, concerns regarding their safety remain paramount in these high-stake domains. Moreover, the development of LLMs is disproportionately focused on English. It remains unclear how these LLMs perform in the context of non-English languages, a gap that is critical for ensuring equity in the real-world use of these systems.This paper provides a framework to investigate the effectiveness of LLMs as multi-lingual dialogue systems for healthcare queries. Our empirically-derived framework XlingEval focuses on three fundamental criteria for evaluating LLM responses to naturalistic human-authored health-related questions: correctness, consistency, and verifiability. Through extensive experiments on four major global languages, including English, Spanish, Chinese, and Hindi, spanning three expert-annotated large health Q&A datasets, and through an amalgamation of algorithmic and human-evaluation strategies, we found a pronounced disparity in LLM responses across these languages, indicating a need for enhanced cross-lingual capabilities. We further propose XlingHealth, a cross-lingual benchmark for examining the multilingual capabilities of LLMs in the healthcare context. Our findings underscore the pressing need to bolster the cross-lingual capacities of these models, and to provide an equitable information ecosystem accessible to all.

Motivation & Objective

• Motivate equitable access to health information by evaluating LLMs beyond English in high-stakes domains.
• Propose a multilingual evaluation framework (XLingEval) focusing on correctness, consistency, and verifiability.
• Create a multilingual health-care benchmark (XLingHealth) spanning four widely spoken languages.
• Assess cross-lingual performance and generalizability across multiple LLMs on real-world health QA datasets.

Proposed method

• Define three core evaluation criteria for health queries: correctness, consistency, and verifiability.
• Develop XLingEval with automated and human evaluation components to compare LLM outputs against expert-ground-truth across languages.
• Build XLingHealth by translating English health QA datasets (HealthQA, LiveQA, MedicationQA) into Hindi, Chinese, and Spanish with medical expert input.
• Conduct multilingual experiments using GPT-3.5 and MedAlpaca-30b, analyzing language disparity across datasets and languages.
• Apply statistical tests (ANOVA, Tukey HSD, t-tests) to determine significance of cross-language performance differences.
• Use multiple similarity metrics (n-gram, BERTScore, Sentence Embedding) and topic models (LDA, HDP) to assess consistency at surface, semantic, and topic levels.
• Evaluate verifiability by treating the model as a detector of correct vs. incorrect claims across datasets.

Figure 1 . We present XLingEval , a comprehensive framework for assessing cross-lingual behaviors of LLMs for high risk domains such as healthcare. We present XLingHealth , a cross-lingual benchmark for healthcare queries.

Experimental results

Research questions

• RQ1 How do LLMs perform on healthcare queries across English, Spanish, Chinese, and Hindi?
• RQ2 Do correctness, consistency, and verifiability exhibit cross-lingual disparities in health Q&A?
• RQ3 Can the XLingEval framework reliably detect multilingual gaps and guide improvements in cross-lingual health information access?
• RQ4 Are multilingual benchmarks like XLingHealth generalizable to other domains and models?

Key findings

• Across four languages, there is a pronounced disparity in correctness, with non-English queries yielding more incorrect responses than English for GPT-3.5.
• GPT-3.5 shows a 5.82x higher likelihood of incorrect answers in non-English vs English for non-English queries in health datasets.
• Consistency analyses show up to 50.5% performance drop in Hindi and 28.3% in Chinese vs English for certain metrics.
• Verifiability is notably weaker in Chinese and Hindi; English and Spanish perform comparatively better (HealthQA: English vs Chinese/Hindi).
• MedAlpaca-30b shows language-disparity patterns that differ from GPT-3.5, highlighting model-dependent cross-lingual behavior.
• ANOVA indicates statistically significant language differences across metrics and models; English-Spanish often closer in performance, while other pairs show larger gaps.

Figure 2 . Evaluation pipelines for correctness, consistency, and verifiability criteria in the XLingEval framework.