[Paper Review] medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs

Mingyi Jia, Junwen Duan+2 more|arXiv (Cornell University)|Jun 20, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

medIKAL proposes a novel framework that integrates knowledge graphs (KGs) as external assistants to large language models (LLMs) for improved clinical diagnosis on electronic medical records (EMRs). By assigning type-based weights to medical entities and using a residual-network-inspired integration method, medIKAL reduces LLM over-reliance on KGs while enhancing diagnostic accuracy, achieving a 36.93 F1 score on a new Chinese EMR dataset.

ABSTRACT

Electronic Medical Records (EMRs), while integral to modern healthcare, present challenges for clinical reasoning and diagnosis due to their complexity and information redundancy. To address this, we proposed medIKAL (Integrating Knowledge Graphs as Assistants of LLMs), a framework that combines Large Language Models (LLMs) with knowledge graphs (KGs) to enhance diagnostic capabilities. medIKAL assigns weighted importance to entities in medical records based on their type, enabling precise localization of candidate diseases within KGs. It innovatively employs a residual network-like approach, allowing initial diagnosis by the LLM to be merged into KG search results. Through a path-based reranking algorithm and a fill-in-the-blank style prompt template, it further refined the diagnostic process. We validated medIKAL's effectiveness through extensive experiments on a newly introduced open-sourced Chinese EMR dataset, demonstrating its potential to improve clinical diagnosis in real-world settings.

Motivation & Objective

• To address the limitations of existing LLM-KG integration methods in EMR diagnosis, particularly over-reliance on external knowledge and poor handling of redundant, complex EMR data.
• To develop a framework that leverages structured knowledge from KGs while preserving and integrating the LLM’s internal medical knowledge.
• To improve diagnostic performance on information-dense and redundant EMRs through entity-type-aware weighting and path-based reranking.
• To introduce a new open-source, high-quality Chinese EMR dataset (CMEMR) to support future research in clinical NLP.
• To minimize hallucinations and enhance fault tolerance by balancing LLM-generated predictions with KG-augmented reasoning.

Proposed method

• Assigns type-based weights to entities in EMRs to prioritize clinically relevant terms (e.g., symptoms, diagnoses) during KG search, improving candidate disease localization.
• Employs a residual-network-like mechanism to first generate an initial diagnosis via the LLM independently, then fuses this with KG search results to reduce over-reliance on external knowledge.
• Applies a path-based reranking algorithm to rank candidate diseases based on semantic paths in the KG, improving diagnostic precision.
• Uses a fill-in-the-blank style prompt template to guide LLMs in error correction and reasoning, enhancing inference quality.
• Preprocesses EMRs via summarization to reduce redundancy before entity linking and KG retrieval.
• Integrates LLM and KG outputs through a late-fusion strategy that preserves useful predictions from the LLM while correcting errors via KG evidence.

Experimental results

Research questions

• RQ1 Can a weighted entity type approach improve the precision of candidate disease retrieval from knowledge graphs in EMR diagnosis?
• RQ2 Does a residual-style integration of LLM and KG outputs reduce over-reliance on external knowledge and preserve useful internal LLM predictions?
• RQ3 To what extent does path-based reranking improve diagnostic ranking performance compared to baseline retrieval methods?
• RQ4 How effective is the fill-in-the-blank prompt template in enhancing LLM reasoning and error correction in clinical diagnosis?
• RQ5 Can the proposed framework maintain high performance on sparse or low-information EMR inputs?

Key findings

• medIKAL achieved an F1 score of 36.93 on the CMEMR dataset, significantly outperforming baseline methods in diagnostic accuracy.
• The ablation study showed that removing the entity-type weighting (w/o ETW) reduced F1 to 34.63, demonstrating its critical role in filtering noise and focusing on relevant entities.
• The residual integration module (w/o RI) caused the largest performance drop (F1: 34.55), confirming its importance in balancing LLM and KG contributions.
• The summarization module (w/o SUM) reduced performance to F1: 36.39, indicating that redundancy reduction improves downstream task efficiency.
• medIKAL retained 78% of useful LLM predictions, significantly higher than baselines, showing reduced over-reliance on KG knowledge.
• Case studies confirmed that medIKAL can correct LLM hallucinations, complement missing knowledge, and guide reasoning, especially in complex diagnostic scenarios.