[Paper Review] DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature

Dawei Li, Yang Shu+11 more|arXiv (Cornell University)|May 8, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

DALK proposes a dynamic co-augmentation framework that synergistically enhances large language models (LLMs) and knowledge graphs (KGs) for Alzheimer’s Disease (AD) question answering. By using LLMs to construct an evolving AD-specific KG from scientific literature and applying a coarse-to-fine, self-aware knowledge retrieval method, DALK improves answer accuracy on a new ADQA benchmark, achieving 72.6% F1, outperforming baselines and demonstrating the value of mutual LLM-KG enhancement in domain-specific biomedical QA.

ABSTRACT

Recent advancements in large language models (LLMs) have achieved promising performances across various applications. Nonetheless, the ongoing challenge of integrating long-tail knowledge continues to impede the seamless adoption of LLMs in specialized domains. In this work, we introduce DALK, a.k.a. Dynamic Co-Augmentation of LLMs and KG, to address this limitation and demonstrate its ability on studying Alzheimer's Disease (AD), a specialized sub-field in biomedicine and a global health priority. With a synergized framework of LLM and KG mutually enhancing each other, we first leverage LLM to construct an evolving AD-specific knowledge graph (KG) sourced from AD-related scientific literature, and then we utilize a coarse-to-fine sampling method with a novel self-aware knowledge retrieval approach to select appropriate knowledge from the KG to augment LLM inference capabilities. The experimental results, conducted on our constructed AD question answering (ADQA) benchmark, underscore the efficacy of DALK. Additionally, we perform a series of detailed analyses that can offer valuable insights and guidelines for the emerging topic of mutually enhancing KG and LLM. We will release the code and data at https://github.com/David-Li0406/DALK.

Motivation & Objective

• To address the limitations of LLMs in capturing long-tail and domain-specific knowledge in Alzheimer’s Disease (AD), particularly due to noisy or irrelevant information in scientific literature.
• To overcome the inefficiency and scalability issues of retraining domain-specific LLMs by leveraging a lightweight, tuning-free co-augmentation framework.
• To improve the quality and relevance of retrieved knowledge for LLM inference by constructing a specialized AD knowledge graph and applying a self-aware retrieval mechanism.
• To establish a new, large-scale AD question-answering benchmark (ADQA) derived from medical exam questions and LLM self-sampling, enabling rigorous evaluation of AD-specific LLM systems.
• To provide actionable insights into effective KG construction and knowledge retrieval strategies for biomedical LLM applications in complex, evolving domains like AD.

Proposed method

• Leverages LLMs to extract and structure knowledge from unstructured AD-related scientific literature, constructing a dynamic, evolving AD-specific knowledge graph (AD-KG) using both pair-wise and generative knowledge construction methods.
• Employs a coarse-to-fine sampling strategy to progressively narrow down candidate knowledge subgraphs from the AD-KG, reducing noise and improving retrieval relevance.
• Introduces a novel self-aware knowledge retrieval approach that dynamically selects the top-k most relevant triples based on semantic alignment with the input query, minimizing irrelevant information.
• Designs a tuning-free framework that avoids retraining LLMs, enabling efficient deployment with large-scale and API-based LLMs by decoupling KG construction and inference augmentation.
• Constructs the ADQA benchmark by filtering millions of samples from general medical QA datasets using a curated keyword list and LLM-based self-sampling, ensuring domain relevance and coverage.
• Uses chain-of-thought prompting in LLM inference, augmented with retrieved KG triples, to improve reasoning and answer accuracy on AD-specific questions.

Figure 1: The overview pipeline of DALK. We first extract structural knowledge from unstructured corpora and construct a domain-specific knowledge graph tailored to AD (Section 3.1 ). Then, we utilize a coarse-to-fine sampling method with a novel self-aware knowledge retrieval approach to select app

Experimental results

Research questions

• RQ1 How effective is the dynamic co-augmentation of LLMs and KGs in improving answer accuracy for Alzheimer’s Disease question-answering tasks compared to standard retrieval-augmented or domain-fine-tuned LLMs?
• RQ2 What impact do different knowledge graph construction methods (pair-wise vs. generative) have on the quality and downstream performance of the resulting AD-KG?
• RQ3 How does the self-aware knowledge retrieval mechanism improve answer accuracy compared to baseline retrieval strategies, especially in the presence of noisy or irrelevant subgraphs?
• RQ4 To what extent does the inclusion of specific AD-related keywords (e.g., APOE, amyloid beta) influence model performance, and which keywords are underrepresented in current QA benchmarks?
• RQ5 Can a lightweight, tuning-free co-augmentation framework effectively enhance LLM performance in a high-stakes, knowledge-intensive domain like Alzheimer’s disease without requiring model fine-tuning?

Key findings

• DALK achieved an F1 score of 72.6% on the ADQA benchmark, significantly outperforming baseline models, including retrieval-augmented and domain-specific LLMs, demonstrating the effectiveness of dynamic co-augmentation.
• The self-aware knowledge retrieval method improved performance by 2.0 points (from 70.6% to 72.6%) compared to the baseline without self-aware retrieval, highlighting its impact on reducing noise and improving relevance.
• Removing specific AD-related keywords such as 'APOE' and 'Amyloid beta' led to notable performance drops (e.g., 73.2% and 73.5% F1 respectively), indicating their critical role in AD knowledge representation.
• The sensitivity analysis revealed that keywords like 'CSF Biomarkers' and 'Neurogenesis' were largely absent from the ADQA benchmark, suggesting a need for targeted data collection to improve benchmark coverage.
• The coarse-to-fine sampling method reduced subgraph size and noise, particularly for longer queries, which otherwise led to larger and noisier subgraphs that hurt performance.
• The ablation study showed that removing 'Alzheimer' or 'Dementia' from the query context caused a slight drop in performance, indicating that these terms are important for grounding the query in the AD domain.

Figure 2: The detailed process of AD-specific KG construction.