[Paper Review] BiMediX: Bilingual Medical Mixture of Experts LLM

Sara Pieri, Sahal Shaji Mullappilly+5 more|arXiv (Cornell University)|Feb 20, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

BiMediX is the first bilingual medical mixture-of-experts LLM designed for seamless, high-performance medical interactions in both English and Arabic. It leverages a semi-automated, human-refined translation pipeline to create BiMed1.3M—a 1.3M-sample bilingual instruction dataset with 632M healthcare-specialized tokens—enabling state-of-the-art performance on English and Arabic benchmarks, outperforming Med42 and Meditron by 2.5% and 4.1% respectively, and Jais-30B by 10% on Arabic tasks, while achieving 8x faster inference.

ABSTRACT

In this paper, we introduce BiMediX, the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic. Our model facilitates a wide range of medical interactions in English and Arabic, including multi-turn chats to inquire about additional details such as patient symptoms and medical history, multiple-choice question answering, and open-ended question answering. We propose a semi-automated English-to-Arabic translation pipeline with human refinement to ensure high-quality translations. We also introduce a comprehensive evaluation benchmark for Arabic medical LLMs. Furthermore, we introduce BiMed1.3M, an extensive Arabic-English bilingual instruction set covering 1.3 Million diverse medical interactions, resulting in over 632 million healthcare specialized tokens for instruction tuning. Our BiMed1.3M dataset includes 250k synthesized multi-turn doctor-patient chats and maintains a 1:2 Arabic-to-English ratio. Our model outperforms state-of-the-art Med42 and Meditron by average absolute gains of 2.5% and 4.1%, respectively, computed across multiple medical evaluation benchmarks in English, while operating at 8-times faster inference. Moreover, our BiMediX outperforms the generic Arabic-English bilingual LLM, Jais-30B, by average absolute gains of 10% on our Arabic medical benchmark and 15% on bilingual evaluations across multiple datasets. Our project page with source code and trained model is available at https://github.com/mbzuai-oryx/BiMediX .

Motivation & Objective

• To develop a bilingual medical LLM capable of seamless, interactive medical conversations in both English and Arabic.
• To address the scarcity of high-quality Arabic medical LLMs by creating a comprehensive, high-fidelity Arabic-English instruction dataset.
• To design a semi-automated, human-refined translation pipeline that ensures linguistic and medical accuracy in low-resource language translation.
• To establish a new benchmark for evaluating Arabic medical LLMs and bilingual medical models.
• To achieve state-of-the-art performance in both English and Arabic medical evaluation tasks without requiring separate fine-tuning per benchmark.

Proposed method

• A semi-automated, iterative translation pipeline was developed to translate English medical text into Arabic, incorporating human verification to ensure high-quality, medically accurate translations.
• BiMed1.3M, a 1.3 million-sample bilingual instruction dataset, was curated, including 250k synthesized multi-turn doctor-patient chats and maintaining a 1:2 Arabic-to-English token ratio.
• The BiMediX model is a mixture-of-experts (MoE) architecture fine-tuned on BiMed1.3M, enabling efficient, high-capacity inference with expert routing per input token.
• A comprehensive Arabic medical evaluation benchmark was constructed to assess model performance in Arabic-specific medical tasks.
• The model was evaluated across multiple English and Arabic medical benchmarks, including MCQA, open-ended QA, and multi-turn dialogues, using standardized metrics.
• Inference efficiency was measured via latency and tokens per second on A100-80GB GPUs, demonstrating 8x faster inference than baseline models.

Experimental results

Research questions

• RQ1 Can a bilingual medical mixture-of-experts LLM achieve state-of-the-art performance in both English and Arabic medical benchmarks without separate fine-tuning per task?
• RQ2 How effective is a semi-automated, human-refined translation pipeline in generating high-quality, medically accurate Arabic translations from English medical text?
• RQ3 To what extent does a large-scale, multilingual instruction dataset like BiMed1.3M improve zero-shot and few-shot generalization in bilingual medical LLMs?
• RQ4 How does BiMediX compare to existing bilingual and monolingual medical LLMs in Arabic medical understanding and multilingual medical reasoning?
• RQ5 What is the impact of bilingual pre-training on Arabic medical model performance compared to monolingual Arabic-only fine-tuning?

Key findings

• BiMediX outperformed Med42 and Meditron by average absolute gains of 2.5% and 4.1% respectively across multiple English medical benchmarks.
• On the Arabic medical benchmark, BiMediX achieved a 10% average performance gain over Jais-30B, a leading generic Arabic-English bilingual LLM.
• In bilingual evaluations, BiMediX surpassed Jais-30B by 15% on average, demonstrating the effectiveness of its bilingual training data.
• BiMediX achieved 8 times faster inference speed than baseline models, measured in tokens per second on A100-80GB GPUs.
• The model demonstrated strong zero-shot generalization across diverse medical tasks, including multi-turn dialogues, MCQA, and open-ended QA, without task-specific fine-tuning.
• The integration of 250k synthesized multi-turn doctor-patient chats in BiMed1.3M significantly enhanced the model’s ability to conduct follow-up inquiries and maintain conversational continuity.