[Paper Review] KG-RAG: Bridging the Gap Between Knowledge and Creativity

Diego Sanmartin|arXiv (Cornell University)|May 20, 2024

Digital Innovation in Industries16 citations

TL;DR

KG-RAG combines Knowledge Graphs with retrieval-augmented generation to reduce hallucinations and improve knowledge-grounded reasoning in LLM-based agents. It introduces a Chain of Explorations (CoE) for KGQA over a learned KG from unstructured text.

ABSTRACT

Ensuring factual accuracy while maintaining the creative capabilities of Large Language Model Agents (LMAs) poses significant challenges in the development of intelligent agent systems. LMAs face prevalent issues such as information hallucinations, catastrophic forgetting, and limitations in processing long contexts when dealing with knowledge-intensive tasks. This paper introduces a KG-RAG (Knowledge Graph-Retrieval Augmented Generation) pipeline, a novel framework designed to enhance the knowledge capabilities of LMAs by integrating structured Knowledge Graphs (KGs) with the functionalities of LLMs, thereby significantly reducing the reliance on the latent knowledge of LLMs. The KG-RAG pipeline constructs a KG from unstructured text and then performs information retrieval over the newly created graph to perform KGQA (Knowledge Graph Question Answering). The retrieval methodology leverages a novel algorithm called Chain of Explorations (CoE) which benefits from LLMs reasoning to explore nodes and relationships within the KG sequentially. Preliminary experiments on the ComplexWebQuestions dataset demonstrate notable improvements in the reduction of hallucinated content and suggest a promising path toward developing intelligent systems adept at handling knowledge-intensive tasks.

Motivation & Objective

• Motivate and address the challenge of factual inaccuracy (hallucinations) and memory limitations in LLM-based agents during knowledge-intensive tasks.
• Propose a KG-RAG pipeline that builds a homogeneous knowledge graph from unstructured text and uses KGQA for grounding reasoning.
• Reduce reliance on latent LLM knowledge by integrating external, updateable knowledge graphs.
• Introduce a novel retrieval algorithm, Chain of Explorations (CoE), to navigate the KG for precise answers.

Proposed method

• Storage: extract (entity, relation, entity) triples from text using a 6-shot prompting LLM and construct triple hypernodes for nested relationships; store in a KG with embeddings in a vector store.
• Retrieval: apply Chain of Explorations (CoE) over the KG, guided by planning, KG lookups (vectorDB and Cypher queries), and evaluation to select relevant paths.
• Answer Generation: generate answers with an LLM constrained to rely only on KG-derived context, using a standard RAG prompt.
• KG Construction Details: define triple hypernodes to model nested structures, enabling multi-layered relationships within a single node; embed all nodes/hypernodes/relations for dense retrieval.
• Experimental Setup: use ComplexWebQuestions dataset, NebulaGraph for KG storage, SentenceTransformer embeddings in Redis, and GPT-4 Turbo 1106-Preview for the LLM; assess with EM, F1, Accuracy, and Hallucination metrics.

Figure 1: shows the three core components of an AI agent: perception, brain, and action. The brain component integrates LLMs for dynamic reasoning and decision-making, alongside KGs for structured knowledge and memory storage.

Experimental results

Research questions

• RQ1 Can KG-RAG improve factual grounding and reduce hallucinations compared to traditional RAG methods on knowledge-intensive tasks?
• RQ2 Does the Chain of Explorations (CoE) retrieval method effectively navigate KGs to support accurate KGQA?
• RQ3 How does KG-RAG performance on ComplexWebQuestions compare to embedding-based RAG approaches in terms of EM, F1, accuracy, and hallucination rate?

Key findings

• KG-RAG achieved EM 19%, F1 25%, accuracy 32%, and hallucination rate 15% on CWQ, indicating improved factual grounding relative to some baselines but lagging top models on exact metrics.
• Compared to Embedding-RAG, KG-RAG has lower EM (19% vs 28%) and F1 (25% vs 37%), and lower accuracy (32% vs 46%), but a notably reduced hallucination rate (15% vs 30%).
• Average Chain of Explorations traversed 4–5 steps to reach answer nodes, illustrating the iterative KG-guided retrieval process.
• The approach demonstrates potential benefits of dynamic, structured knowledge (KGs) over purely dense retrieval for complex, multi-hop questions, albeit with room for efficiency and coverage improvements.
• Limitations include data quality and cost constraints in KG construction, and some questions where start nodes were not identified due to snippet selection.

(a)