[Paper Review] Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking

Qinyong Wang, Zhenxiang Gao+1 more|arXiv (Cornell University)|Jul 3, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a two-stage in-context learning (ICL) framework for biomedical concept linking that uses dense retrieval followed by LLM-based re-ranking, achieving 90.1% accuracy in disease normalization and 94.7% in chemical normalization—competitive with supervised methods—without any fine-tuning or labeled data.

ABSTRACT

The biomedical field relies heavily on concept linking in various areas such as literature mining, graph alignment, information retrieval, question-answering, data, and knowledge integration. Although large language models (LLMs) have made significant strides in many natural language processing tasks, their effectiveness in biomedical concept mapping is yet to be fully explored. This research investigates a method that exploits the in-context learning (ICL) capabilities of large models for biomedical concept linking. The proposed approach adopts a two-stage retrieve-and-rank framework. Initially, biomedical concepts are embedded using language models, and then embedding similarity is utilized to retrieve the top candidates. These candidates' contextual information is subsequently incorporated into the prompt and processed by a large language model to re-rank the concepts. This approach achieved an accuracy of 90.% in BC5CDR disease entity normalization and 94.7% in chemical entity normalization, exhibiting a competitive performance relative to supervised learning methods. Further, it showed a significant improvement, with an over 20-point absolute increase in F1 score on an oncology matching dataset. Extensive qualitative assessments were conducted, and the benefits and potential shortcomings of using large language models within the biomedical domain were discussed. were discussed.

Motivation & Objective

• Address the limitations of supervised methods in biomedical concept linking, such as high annotation cost, data expiration, and poor generalization across tasks.
• Explore whether in-context learning (ICL) in large language models (LLMs) can enable zero-shot, adaptable biomedical concept linking without task-specific fine-tuning.
• Develop a generalized, robust framework for concept linking that handles ambiguity, rare diseases, and heterogeneous data sources.
• Evaluate the interpretability and reasoning quality of LLMs in biomedical linking tasks, particularly for complex or rare disease subtypes.
• Assess the feasibility and performance of LLM-based linking using both closed-source (GPT-4) and open-source (Llama) models in real-world biomedical datasets.

Proposed method

• Employ a two-stage framework: first, retrieve candidate biomedical concepts using dense vector embeddings from a language model.
• Compute embedding similarity to identify top-k candidate concepts for each text mention.
• Construct in-context learning prompts by incorporating the context of each candidate and the original mention into the LLM input.
• Use the LLM (e.g., GPT-4 or Llama) to re-rank candidates based on contextual relevance and semantic alignment.
• Leverage the LLM’s in-context learning capability to perform zero-shot reasoning without model fine-tuning.
• Incorporate external knowledge such as concept descriptions and abbreviation dictionaries to improve performance on rare or ambiguous entities.

Figure 1: Overview of the proposed methodology and comparison with previous methods.

Experimental results

Research questions

• RQ1 Can in-context learning in LLMs effectively perform biomedical concept linking without any fine-tuning or labeled data?
• RQ2 How does the performance of LLMs like GPT-4 and Llama compare to supervised methods in disease and chemical entity normalization tasks?
• RQ3 To what extent does the reasoning process of LLMs in concept linking reflect accurate, context-aware logic rather than superficial keyword matching?
• RQ4 What are the key failure modes of LLM-based concept linking, particularly for rare diseases or ambiguous abbreviations?
• RQ5 How does the inclusion of external knowledge (e.g., descriptions, abbreviation lists) affect the robustness and accuracy of LLM-based linking?

Key findings

• The proposed ICL-based framework achieved 90.1% F1 in BC5CDR disease entity normalization and 94.7% in chemical entity normalization, outperforming baseline zero-shot methods and approaching supervised baselines.
• On an oncology matching dataset, the method showed an over 20-point absolute increase in F1 score compared to baseline methods, demonstrating strong generalization and robustness.
• GPT-4 demonstrated superior reasoning and accuracy, especially for rare disease subtypes and complex concept descriptions, while Llama 13B struggled with both accuracy and logical consistency.
• The reasoning process of GPT-4 was generally aligned with provided context and descriptions, enhancing interpretability, whereas Llama sometimes produced incorrect reasoning despite correct predictions.
• The framework exhibited failure cases when multiple concepts shared the same name or when abbreviations were ambiguous, indicating the need for external knowledge integration.
• Inference with GPT-4 was costly ($150 for 3,700 pairs), and even local LLMs like Llama 13B were slow (103 ms/token), highlighting scalability and hardware limitations for widespread adoption.

Figure 2: Workflow of the proposed methodology for exploring the in-context learning ability of large language models for biomedical concept linking.