[Paper Review] Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People

Xinyin Wang, Nuo Chen+10 more|arXiv (Cornell University)|Mar 6, 2024

Artificial Intelligence in Healthcare and Education10 citations

TL;DR

Apollo proposes lightweight multilingual medical LLMs (0.5B–7B) trained on ApolloCorpora to cover English, Chinese, Hindi, Spanish, French, and Arabic, introduces XMedBench for evaluation, and shows proxy tuning can extend multilingual medical capabilities to larger models without fine-tuning.

ABSTRACT

Despite the vast repository of global medical knowledge predominantly being in English, local languages are crucial for delivering tailored healthcare services, particularly in areas with limited medical resources. To extend the reach of medical AI advancements to a broader population, we aim to develop medical LLMs across the six most widely spoken languages, encompassing a global population of 6.1 billion. This effort culminates in the creation of the ApolloCorpora multilingual medical dataset and the XMedBench benchmark. In the multilingual medical benchmark, the released Apollo models, at various relatively-small sizes (i.e., 0.5B, 1.8B, 2B, 6B, and 7B), achieve the best performance among models of equivalent size. Especially, Apollo-7B is the state-of-the-art multilingual medical LLMs up to 70B. Additionally, these lite models could be used to improve the multi-lingual medical capabilities of larger models without fine-tuning in a proxy-tuning fashion. We will open-source training corpora, code, model weights and evaluation benchmark.

Motivation & Objective

• Democratize Medical AI by extending multilingual medical capabilities to the six most spoken languages (English, Chinese, Hindi, Spanish, French, Arabic).
• Build high-quality multilingual medical data (ApolloCorpora) and an evaluation benchmark (XMedBench) to assess medical knowledge across languages.
• Show that lightweight models (0.5B–7B) can outperform equivalent-sized peers and can enhance larger models via proxy tuning without exposing private medical data.
• Demonstrate that multilingual training and data rewriting strategies can improve cross-language medical understanding while analyzing potential multilingual training risks.

Proposed method

• Create ApolloCorpora by collecting high-quality medical data in six languages from books, papers, encyclopedias, guidelines, dialogues, exams, and web sources.
• Rewrite portions of pre-training data into QA pairs using ChatGPT to facilitate a smoother transition from pre-training to instruction tuning.
• Apply adaptive, priority-based data sampling to mix pre-training and instruction-tuning data.
• Train Apollo models at 0.5B, 1.8B, 2B, 6B, and 7B parameters, and perform multilingual training and language-specific training comparisons.
• Evaluate with XMedBench, a multilingual medical knowledge assessment, using prompts that mimic real medical exam questions across languages.
• Propose and evaluate ProxyTuning to align larger models with small, fine-tuned medical models without direct fine-tuning on private medical data.

Experimental results

Research questions

• RQ1 Does multilingual training of medical LLMs yield consistently better performance across languages than language-specific models or simple weight averaging?
• RQ2 How does rewriting pre-training data into QA pairs affect downstream multilingual medical performance?
• RQ3 Can proxy tuning extend multilingual medical capabilities to larger general LLMs without exposing private medical corpora to centralized training?
• RQ4 What are the language-specific strengths and gaps (e.g., Arabic and Hindi) in multilingual medical LLMs, and how can mix training mitigate them?
• RQ5 What is the impact of mix/training strategies on long-tail languages and cross-language knowledge transfer?

Key findings

• Multilingual training yields average performance improvements over language-specific models across all languages examined.
• Apollo-7B achieves state-of-the-art performance among multilingual medical LLMs of comparable size, and Apollo-7B matches GPT-3.5 in overall performance at its size.
• Arabic and Hindi show weaker performance relative to other languages, highlighting multilingual gaps.
• Rewriting pre-training data into QA pairs alone does not hurt overall performance, and a smooth transition after mix training yields better results for non-mainstream languages.
• Mix training (combining multilingual data) substantially improves multilingual medical capabilities compared to monolingual or single-language training.
• ProxyTuning enables larger models to gain multilingual medical abilities using the small Apollo models as a guide, without fine-tuning the larger model on private medical data.