[Paper Review] Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes

Vahan Arsenyan, Bughdaryan, Spartak+3 more|arXiv (Cornell University)|Jan 29, 2023

Artificial Intelligence in HealthcareHealth Professions3 citations

TL;DR

This paper proposes an end-to-end method using large language models (LLMs) to automatically construct biomedical knowledge graphs from electronic medical record (EMR) notes, focusing on relations between diseases, treatments, factors, and manifestations. It evaluates 12 LLMs across architectures, finds decoder-only models require guided prompting for structured output, and demonstrates the approach on age-related macular degeneration with a manually annotated dataset.

ABSTRACT

The automatic construction of knowledge graphs (KGs) is an important research area in medicine, with far-reaching applications spanning drug discovery and clinical trial design. These applications hinge on the accurate identification of interactions among medical and biological entities. In this study, we propose an end-to-end machine learning solution based on large language models (LLMs) that utilize electronic medical record notes to construct KGs. The entities used in the KG construction process are diseases, factors, treatments, as well as manifestations that coexist with the patient while experiencing the disease. Given the critical need for high-quality performance in medical applications, we embark on a comprehensive assessment of 12 LLMs of various architectures, evaluating their performance and safety attributes. To gauge the quantitative efficacy of our approach by assessing both precision and recall, we manually annotate a dataset provided by the Macula and Retina Institute. We also assess the qualitative performance of LLMs, such as the ability to generate structured outputs or the tendency to hallucinate. The results illustrate that in contrast to encoder-only and encoder-decoder, decoder-only LLMs require further investigation. Additionally, we provide guided prompt design to utilize such LLMs. The application of the proposed methodology is demonstrated on age-related macular degeneration.

Motivation & Objective

• To develop an end-to-end machine learning solution for automatic biomedical knowledge graph construction from EMR notes.
• To evaluate the performance and safety of 12 diverse LLM architectures in clinical relation extraction tasks.
• To address the challenge of structured output generation from decoder-only LLMs in medical NLP.
• To provide a guided prompt design framework to enhance the utility of decoder-only LLMs for KG construction.
• To demonstrate the method’s applicability on age-related macular degeneration using a manually annotated dataset.

Proposed method

• Utilizes a diverse set of 12 LLMs with varying architectures—encoder-only, encoder-decoder, and decoder-only—for relation extraction from EMR notes.
• Employs a manually annotated, private dataset from the Macula and Retina Institute to benchmark performance on clinical relation extraction.
• Applies instruction-based and few-shot prompting strategies to improve structured output generation, especially for decoder-only models.
• Focuses on extracting directed relationships of type 'coexists_with' between medical entities: diseases, treatments, factors, and manifestations.
• Conducts both quantitative evaluation (precision, recall) and qualitative analysis (hallucination, output structure) of LLM behavior.
• Uses a pipeline that processes EMR notes to identify entities and relations, then constructs a directed knowledge graph with nodes and labeled edges.

Experimental results

Research questions

• RQ1 Which LLM architectures perform best in extracting structured medical relations from EMR notes?
• RQ2 How do different LLM architectures compare in terms of precision, recall, and hallucination rates in clinical relation extraction?
• RQ3 Can guided prompting effectively enable decoder-only LLMs to generate structured outputs suitable for knowledge graph construction?
• RQ4 What are the safety and reliability concerns when deploying LLMs in clinical knowledge graph construction?
• RQ5 How effective is the proposed method in constructing knowledge graphs for specific conditions like age-related macular degeneration?

Key findings

• Decoder-only LLMs require guided prompting to produce structured outputs suitable for knowledge graph construction, unlike encoder-only and encoder-decoder models.
• FLAN-T5-XXL and FLAN-UL2 models showed strong performance with instruction-based and few-shot prompting, achieving correct relation extraction in multiple test cases.
• The study found that models fine-tuned on public datasets like MIMIC-III may be biased or less generalizable, so a private, manually annotated dataset was used for evaluation.
• Guided prompting significantly improved the reliability and consistency of relation extraction, especially for decoder-only models.
• The method successfully constructed a knowledge graph for age-related macular degeneration, identifying key relations such as 'AREDS and WACS vitamins' coexisting with 'myopic macular degeneration'.
• Qualitative analysis revealed that hallucination and unstructured output were common issues in decoder-only models without proper prompting.