[Paper Review] KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models

Jiho Kim, Yeonsu Kwon+2 more|arXiv (Cornell University)|Oct 17, 2023

Topic ModelingComputer Science3 citations

TL;DR

KG-GPT is a novel framework that leverages large language models (LLMs) for reasoning on knowledge graphs (KGs) through a three-stage process: sentence segmentation, graph retrieval, and inference. It achieves competitive performance on KG-based fact verification and question answering benchmarks, outperforming several fully-supervised models and demonstrating robustness across multi-hop reasoning tasks.

ABSTRACT

While large language models (LLMs) have made considerable advancements in understanding and generating unstructured text, their application in structured data remains underexplored. Particularly, using LLMs for complex reasoning tasks on knowledge graphs (KGs) remains largely untouched. To address this, we propose KG-GPT, a multi-purpose framework leveraging LLMs for tasks employing KGs. KG-GPT comprises three steps: Sentence Segmentation, Graph Retrieval, and Inference, each aimed at partitioning sentences, retrieving relevant graph components, and deriving logical conclusions, respectively. We evaluate KG-GPT using KG-based fact verification and KGQA benchmarks, with the model showing competitive and robust performance, even outperforming several fully-supervised models. Our work, therefore, marks a significant step in unifying structured and unstructured data processing within the realm of LLMs.

Motivation & Objective

• To bridge the gap between unstructured text reasoning and structured knowledge graph (KG) reasoning using large language models (LLMs).
• To address the lack of general frameworks for LLM-based reasoning on KGs, especially for complex, multi-hop tasks.
• To develop a unified, end-to-end framework that supports diverse KG-based tasks such as fact verification and question answering.
• To enhance LLM reasoning by retrieving relevant sub-graphs (evidence graphs) rather than relying solely on path-based reasoning.
• To demonstrate that in-context learning with LLMs can effectively handle structured knowledge without fine-tuning, maintaining robustness across increasing hop counts.

Proposed method

• Sentence Segmentation: Breaks down input claims or questions into sub-sentences, each aligned with a single triple (head-relation-tail) to simplify reasoning.
• Graph Retrieval: Uses LLMs to map sub-sentence relations to potential relations in the KG, retrieving a candidate pool of relations for evidence graph construction.
• Evidence Graph Construction: Builds a sub-KG (evidence graph) using retrieved relations and entities from the input, forming a logical basis for inference.
• Inference: Employs the retrieved sub-KG as context in a prompt to guide the LLM in generating logical conclusions, such as claim validation or answer generation.
• In-context learning is used throughout all stages to maximize the LLM’s reasoning capability without fine-tuning.
• The framework supports both 1-hop and multi-hop reasoning by dynamically retrieving and combining relevant graph components.

Experimental results

Research questions

• RQ1 Can LLMs effectively perform complex reasoning on knowledge graphs without fine-tuning, using only in-context learning?
• RQ2 How does KG-GPT’s sub-KG retrieval strategy compare to path-based reasoning in terms of reasoning accuracy and robustness?
• RQ3 Does KG-GPT maintain strong performance across increasing numbers of hops in multi-hop reasoning tasks?
• RQ4 Can KG-GPT outperform fully-supervised models on KG-based fact verification and question answering benchmarks?
• RQ5 How sensitive is KG-GPT’s performance to the number of candidate relations retrieved (k) in the graph retrieval stage?

Key findings

• KG-GPT achieved competitive performance on the FactKG benchmark, with a 72.68% accuracy when retrieving the top 5 relations, outperforming several fully-supervised models.
• On the MetaQA benchmark, KG-GPT achieved 96.3% accuracy for 1-hop, 94.4% for 2-hop, and 94.0% for 3-hop questions—showing minimal performance drop with increasing hop count.
• The framework maintained high robustness across all hop levels, indicating effective handling of complex, multi-hop reasoning tasks.
• Performance remained stable across different k-values (1, 3, 5) in relation retrieval, with minimal degradation in accuracy and only moderate increases in retrieved triples.
• Qualitative results confirmed that KG-GPT correctly retrieved relevant evidence graphs and generated accurate predictions for diverse claims and questions.
• The average number of retrieved triples increased with k, but performance did not significantly decline, suggesting efficient and scalable sub-KG construction.