[Paper Review] Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries

Jiageng Wu, Xian Wu+6 more|arXiv (Cornell University)|May 17, 2023

Educational Technology and Pedagogy4 citations

TL;DR

This study introduces the Knowledge and Few-shot Enhancement in-context Learning (KFE) framework to enhance large language models (LLMs) in non-English medical settings by integrating external clinical knowledge and few-shot exemplars. KFE significantly boosts performance on the Chinese National Medical Licensing Examination (CNMLE-2022), enabling GPT-4 to achieve 82.59/100—surpassing the human average (68.70) and allowing even a 13B-parameter model to pass the exam, demonstrating robustness and scalability in low-resource linguistic contexts.

ABSTRACT

$ extbf{Objectives}$: Large Language Models (LLMs) such as ChatGPT and Med-PaLM have excelled in various medical question-answering tasks. However, these English-centric models encounter challenges in non-English clinical settings, primarily due to limited clinical knowledge in respective languages, a consequence of imbalanced training corpora. We systematically evaluate LLMs in the Chinese medical context and develop a novel in-context learning framework to enhance their performance. $ extbf{Materials and Methods}$: The latest China National Medical Licensing Examination (CNMLE-2022) served as the benchmark. We collected 53 medical books and 381,149 medical questions to construct the medical knowledge base and question bank. The proposed Knowledge and Few-shot Enhancement In-context Learning (KFE) framework leverages the in-context learning ability of LLMs to integrate diverse external clinical knowledge sources. We evaluated KFE with ChatGPT(GPT3.5), GPT4, Baichuan2(BC2)-7B, and BC2-13B in CNMLE-2022 and investigated the effectiveness of different pathways for incorporating LLMs with medical knowledge from 7 perspectives. $ extbf{Results}$: Directly applying ChatGPT failed to qualify for the CNMLE-2022 at a score of 51. Cooperated with the KFE, the LLMs with varying sizes yielded consistent and significant improvements. The ChatGPT's performance surged to 70.04 and GPT-4 achieved the highest score of 82.59. This surpasses the qualification threshold (60) and exceeds the average human score of 68.70. It also enabled a smaller BC2-13B to pass the examination, showcasing the great potential in low-resource settings. $ extbf{Conclusion}$: By synergizing medical knowledge through in-context learning, LLM can extend clinical insight beyond language barriers, significantly reducing language-related disparities of LLM applications and ensuring global benefit in healthcare.

Motivation & Objective

• To address the performance gap of English-centric LLMs in non-English clinical settings due to imbalanced training data and limited multilingual clinical knowledge.
• To develop a scalable, training-free framework that enhances LLMs using in-context learning with external medical knowledge and exemplar cases.
• To evaluate the effectiveness of diverse knowledge integration pathways in improving LLM performance on a high-stakes medical examination in Chinese.
• To demonstrate that smaller, low-resource LLMs can achieve expert-level performance when augmented with curated clinical knowledge and few-shot examples.
• To reduce language-related disparities in AI-driven healthcare by enabling global accessibility of LLMs across linguistic and resource-limited settings.

Proposed method

• Constructed a comprehensive medical knowledge base from 53 medical textbooks and a question bank of 381,149 questions aligned with the CNMLE-2022.
• Designed the KFE framework to retrieve relevant clinical knowledge and similar past questions to form a context-aware prompt for in-context learning.
• Integrated specific instructions, retrieved knowledge, exemplar cases, and the target question into a single prompt to guide LLM reasoning.
• Leveraged the in-context learning capability of LLMs to dynamically adapt to new inputs without fine-tuning or retraining.
• Evaluated multiple LLMs—including GPT-3.5, GPT-4, Baichuan2-7b, and Baichuan2-13b—under the KFE framework on the CNMLE-2022 benchmark.
• Systematically analyzed seven distinct pathways for knowledge integration to identify the most effective configurations for performance enhancement.

Figure 1: Workflow of knowledge and few-shot enhancement framework (KFE) . Given the question stem and five candidate options, KFE retrieves the relevant knowledge from a comprehensive knowledge base and identifies similar questions from a historical question bank. The integrated prompt consists of

Experimental results

Research questions

• RQ1 Can external clinical knowledge and few-shot exemplars significantly improve LLM performance on a high-stakes medical licensing exam in a non-English language like Chinese?
• RQ2 How does the KFE framework enhance the clinical reasoning of LLMs without requiring model fine-tuning or retraining?
• RQ3 What are the most effective pathways for integrating diverse medical knowledge sources into LLMs to maximize performance gains?
• RQ4 Can smaller, low-resource LLMs achieve expert-level performance on a medical exam when augmented with curated knowledge and few-shot examples?
• RQ5 To what extent can the KFE framework reduce linguistic and resource-based disparities in AI-driven healthcare applications?

Key findings

• The KFE framework enabled GPT-4 to achieve a score of 82.59 on the CNMLE-2022, surpassing the human average score of 68.70 and exceeding the 60-point qualification threshold.
• ChatGPT (GPT-3.5) improved from a failing score of 51 to 70.04 under KFE, demonstrating consistent and significant performance gains.
• The Baichuan2-13B model, despite being smaller and less pre-trained for medical tasks, passed the CNMLE-2022 after integration with KFE, highlighting its potential in low-resource settings.
• The framework showed robust performance across LLMs of varying sizes, from 7 billion to 1.7 trillion parameters, indicating broad scalability and generalization.
• Different knowledge integration pathways yielded varying performance improvements, with the optimal configuration combining knowledge retrieval, exemplar selection, and structured prompting.
• The study confirms that in-context learning with external knowledge can effectively extend clinical insight beyond language boundaries, reducing disparities in AI healthcare access.

Figure 2: Four strategies of few-shot enhancement.