[Paper Review] Introducing L2M3, A Multilingual Medical Large Language Model to Advance Health Equity in Low-Resource Regions

Agasthya Gangavarapu|arXiv (Cornell University)|Apr 11, 2024

Chronic Disease Management Strategies5 citations

TL;DR

This paper presents the Uheal L2M3 system, a modular multilingual medical LLM augmented with machine translation and guardrails to assist Community Health Workers in LMICs, aiming to improve medical accuracy, safety, and accessibility.

ABSTRACT

Addressing the imminent shortfall of 10 million health workers by 2030, predominantly in Low- and Middle-Income Countries (LMICs), this paper introduces an innovative approach that harnesses the power of Large Language Models (LLMs) integrated with machine translation models. This solution is engineered to meet the unique needs of Community Health Workers (CHWs), overcoming language barriers, cultural sensitivities, and the limited availability of medical dialog datasets. I have crafted a model that not only boasts superior translation capabilities but also undergoes rigorous fine-tuning on open-source datasets to ensure medical accuracy and is equipped with comprehensive safety features to counteract the risks of misinformation. Featuring a modular design, this approach is specifically structured for swift adaptation across various linguistic and cultural contexts, utilizing open-source components to significantly reduce healthcare operational costs. This strategic innovation markedly improves the accessibility and quality of healthcare services by providing CHWs with contextually appropriate medical knowledge and diagnostic tools. This paper highlights the transformative impact of this context-aware LLM, underscoring its crucial role in addressing the global healthcare workforce deficit and propelling forward healthcare outcomes in LMICs.

Motivation & Objective

• Address the imminent global health worker shortfall by enabling CHWs in LMICs to access contextually relevant medical knowledge and diagnostics.
• Develop a modular, culturally localized LLM system with robust safety features to counter misinformation.
• Leverage open-source components to reduce healthcare operating costs and enhance scalability across languages and regions.
• Focus training on DALY-driven health priorities (IHD, LRIs, neonatal care) to maximize impact on burden of disease.

Proposed method

• Curate a domain-adaptive medical corpus totaling 930 million tokens from clinical guidelines, regional guidelines, medical dialogues, and FAERS adverse events data.
• Fine-tune open-source Meditron 70B for medical domain adaptation on 520 million medically relevant tokens using AdamW, 64 batch size, learning rate 2e-5 for 2 epochs on an A100-80GB×8 GPU cluster with AWQ quantization.
• Fine-tune the Meta Seamless M4T v2 Large translation model on a parallel medical corpus to enable bidirectional English↔Telugu, English↔Hindi, English↔Arabic, and English↔Swahili translations.
• Translate non-English medical dialog datasets (Chinese-origin data translated via Azure AI Translator) and apply post-editing to improve colloquial fluency and cultural relevance.
• Implement a two-stage integrated system: (i) translation to English for L2M3 processing, (ii) L2M3 response, (iii) translation back to local language, with NeMo Guardrails ensuring safety, relevance, and jailbreaking resistance.
• Evaluate both domain-adapted model performance and translation accuracy using bespoke rural-CHW evaluation datasets validated by GPT-4 and Claude Opus APIs.

Figure 1: Data Acquisition and Standardization

Experimental results

Research questions

• RQ1 Can a multilingual medical LLM, when coupled with translation and guardrails, provide accurate, safe, and culturally appropriate guidance to CHWs in LMICs?
• RQ2 What is the impact of domain-adaptive fine-tuning on medical diagnostic and care-delivery capabilities in low-resource languages?
• RQ3 How does integrating a translation model with a domain-tuned medical LM affect translation accuracy and error propagation in clinical dialogues?
• RQ4 To what extent can this modular, open-source stack reduce costs and improve health outcomes in rural LMIC settings?

Key findings

• Fine-tuning yields substantial gains in translation quality across languages after domain adaptation (e.g., Telugu BLEU 75.6→82.4, Hindi 73.4→83.1, Swahili 45.8→48.1, Arabic 68.5→80.5; English→Telugu 59.1→81.7, English→Hindi 62.1→83.3, English→Swahili 32.4→40.1, English→Arabic 54.3→78.9).
• Integrated MT+LM system shows potential improvement over standalone LLMs, with preliminary Telugu translation accuracy around 0.48 (0.71×0.675) and other language entries indicating similar benefits from domain tuning and cross-language embedding strategies.
• Domain-adapted L2M3 performance evaluated via two bespoke rural-CHW datasets and validated by GPT-4 and Claude Opus APIs; integrated L2M3 system outperformed GPT-3.5 on Reasoning Hallucination Tests (RHT).
• Translation-aware design revealed context-sensitive challenges (gender/age, COPD terminology) affecting translations; highlights need for contextual localization and ongoing post-editing for medical vocabularies in target languages.
• The system employs NeMo Guardrails to monitor prompts/outputs for safety, reduce hallucinations, and prevent unsafe behavior, without modifying user inputs.

Figure 2: Comparative Performance of GPT-4, Llama 2, Biomistral, and Meditron