[Paper Review] Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Mufei Li, Siqi Miao+1 more|arXiv (Cornell University)|Oct 28, 2024

Topic Modeling4 citations

TL;DR

SubgraphRAG proposes a lightweight, efficient retrieval-augmented generation framework that retrieves flexible subgraphs from knowledge graphs using a parallel triple-scoring mechanism with directional structural encoding, enabling high-quality, explainable reasoning with minimal LLM calls. It achieves state-of-the-art accuracy on WebQSP and CWQ benchmarks—outperforming prior methods in hallucination reduction and truth grounding—while maintaining efficiency and scalability across small (Llama3.1-8B) and large (GPT-4o) models without fine-tuning.

ABSTRACT

Large Language Models (LLMs) demonstrate strong reasoning abilities but face limitations such as hallucinations and outdated knowledge. Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) addresses these issues by grounding LLM outputs in structured external knowledge from KGs. However, current KG-based RAG frameworks still struggle to optimize the trade-off between retrieval effectiveness and efficiency in identifying a suitable amount of relevant graph information for the LLM to digest. We introduce SubgraphRAG, extending the KG-based RAG framework that retrieves subgraphs and leverages LLMs for reasoning and answer prediction. Our approach innovatively integrates a lightweight multilayer perceptron with a parallel triple-scoring mechanism for efficient and flexible subgraph retrieval while encoding directional structural distances to enhance retrieval effectiveness. The size of retrieved subgraphs can be flexibly adjusted to match the query's need and the downstream LLM's capabilities. This design strikes a balance between model complexity and reasoning power, enabling scalable and generalizable retrieval processes. Notably, based on our retrieved subgraphs, smaller LLMs like Llama3.1-8B-Instruct deliver competitive results with explainable reasoning, while larger models like GPT-4o achieve state-of-the-art accuracy compared with previous baselines -- all without fine-tuning. Extensive evaluations on the WebQSP and CWQ benchmarks highlight SubgraphRAG's strengths in efficiency, accuracy, and reliability by reducing hallucinations and improving response grounding.

Motivation & Objective

• To address the trade-off between retrieval efficiency and effectiveness in knowledge-graph-based retrieval-augmented generation (RAG).
• To reduce hallucinations and improve answer grounding by retrieving flexible, relevant subgraphs instead of fixed structures like paths.
• To enable high-performance reasoning with minimal LLM calls by optimizing retrieved context size and relevance.
• To support both small and large LLMs effectively without fine-tuning, ensuring scalability and generalizability.
• To enhance explainability by providing traceable, evidence-based reasoning through retrieved subgraphs.

Proposed method

• Introduces a lightweight multilayer perceptron with parallel triple-scoring to efficiently retrieve relevant subgraphs from knowledge graphs.
• Encodes directional structural distances between entities to improve retrieval relevance and capture multi-hop relationships.
• Dynamically adjusts subgraph size based on query complexity and LLM context window constraints to balance information richness and processing capacity.
• Uses a single LLM call per query for reasoning, avoiding the latency and cost of multiple LLM calls common in existing methods.
• Employs a retrieval-augmented prompting strategy where retrieved subgraphs serve as context for unfine-tuned LLMs to generate answers and explanations.
• Supports both small (e.g., Llama3.1-8B-Instruct) and large (e.g., GPT-4o) LLMs, enabling performance scalability without model fine-tuning.

Figure 1: The framework of SubgraphRAG. Retrieved subgraphs consist of relevant triples that are extracted in parallel. Retrieved subgraphs are flexible in their forms and their sizes. In the above example, the relevant subgraph has flexible and complex forms (neither trees nor paths).

Experimental results

Research questions

• RQ1 How can subgraph retrieval be made both efficient and effective for complex, multi-hop reasoning in knowledge graph-based RAG?
• RQ2 Can a lightweight, non-iterative retriever outperform LLM-driven or GNN-based retrieval methods in accuracy and efficiency?
• RQ3 To what extent does subgraph flexibility improve reasoning performance compared to fixed-structure retrieval (e.g., paths)?
• RQ4 How does the number of retrieved triples affect LLM performance, and how can it be optimized for different model sizes?
• RQ5 Can a retrieval-augmented framework reduce hallucinations and improve truth grounding without fine-tuning?

Key findings

• SubgraphRAG achieves state-of-the-art accuracy on WebQSP and CWQ benchmarks, outperforming previous baselines without model fine-tuning.
• On the CWQ-sub dataset, SubgraphRAG reduces non-retrieval (NR) answers—indicating hallucination—by over 10 percentage points compared to RoG, dropping from 10.3% to 1.0%.
• For questions lacking answers in the KG, SubgraphRAG increases refusal rates from 2% to 19% on WebQSP and from 7% to 29% on CWQ, showing stronger truthfulness and reduced hallucination.
• Llama3.1-8B-Instruct achieves competitive performance with SubgraphRAG, but performance declines when retrieving more than ~100 triples, indicating sensitivity to irrelevant context.
• GPT-4o-mini shows consistent performance improvement with more retrieved triples, demonstrating greater robustness to noisy or redundant context.
• Replacing the retriever with baseline methods (e.g., RoG, G-Retriever) significantly degrades performance, confirming that the key gains stem from the proposed retriever design.

Figure 2: The prompt used in SubgraphRAG. Concrete examples can be found in Appendix D ).