[Paper Review] keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM

Chaojie Wang, Yishi Xu+6 more|arXiv (Cornell University)|Dec 31, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Keqing, a knowledge-based question answering framework that uses structured knowledge graph reasoning paths as natural chain-of-thought (CoT) mentors to guide large language models (LLMs) in solving complex questions step-by-step. By decomposing questions into sub-questions via templates, retrieving relevant knowledge graph triplets, and reasoning over candidate entities, Keqing significantly improves answer reliability and interpretability, achieving state-of-the-art performance on MetaQA (98.4 Hits@1 on 1-hop, 99.6 on 3-hop) and WebQSP (74.9 F1).

ABSTRACT

Large language models (LLMs) have exhibited remarkable performance on various natural language processing (NLP) tasks, especially for question answering. However, in the face of problems beyond the scope of knowledge, these LLMs tend to talk nonsense with a straight face, where the potential solution could be incorporating an Information Retrieval (IR) module and generating response based on these retrieved knowledge. In this paper, we present a novel framework to assist LLMs, such as ChatGPT, to retrieve question-related structured information on the knowledge graph, and demonstrate that Knowledge-based question answering (Keqing) could be a nature Chain-of-Thought (CoT) mentor to guide the LLM to sequentially find the answer entities of a complex question through interpretable logical chains. Specifically, the workflow of Keqing will execute decomposing a complex question according to predefined templates, retrieving candidate entities on knowledge graph, reasoning answers of sub-questions, and finally generating response with reasoning paths, which greatly improves the reliability of LLM's response. The experimental results on KBQA datasets show that Keqing can achieve competitive performance and illustrate the logic of answering each question.

Motivation & Objective

• To address hallucination in LLMs during question answering by grounding responses in factual knowledge from knowledge graphs.
• To improve the reliability and interpretability of LLM-generated answers in complex question-answering tasks.
• To explore whether knowledge-based question answering can naturally serve as a chain-of-thought mentor for LLMs.
• To develop a scalable, automated framework that guides LLMs through logical reasoning paths using pre-collected knowledge graph chains.
• To outperform existing retrieval-augmented and text-to-SQL-based LLM methods in KBQA benchmarks.

Proposed method

• Decomposes complex questions into sub-questions using predefined natural language templates, making logical decomposition easier for LLMs to learn.
• Employs a knowledge retrieval module that retrieves relevant triplets from a knowledge graph based on sub-questions, ensuring high-quality, factual context.
• Uses a candidate reasoning module to iteratively select correct entities across sub-questions, leveraging dependency between decomposed components.
• Generates final responses with explicit reasoning paths, enhancing interpretability and traceability of the answer process.
• Treats pre-collected logical chains from knowledge graphs as natural chain-of-thought supervision, avoiding hand-crafted or SQL-based CoT construction.
• Supports both LLaMA and ChatGPT as the LLM backbone, enabling evaluation across different model capabilities.

Figure 1: The workflow of Keqing applied for KBQA mainly consists of four stages: #1 Question Decomposition : decompose a complex question into several sub-questions according to predefined question templates; #2 Knowledge Retrieval : retrieve candidate entities on the knowledge graph by aligning de

Experimental results

Research questions

• RQ1 Can knowledge-based question answering on a knowledge graph serve as a natural chain-of-thought mentor for LLMs?
• RQ2 Does using natural language templates for question decomposition improve LLM reasoning performance compared to direct logical chain generation?
• RQ3 How does the number of retrieved question templates per sub-question affect the overall KBQA performance?
• RQ4 Can the framework achieve state-of-the-art results while maintaining interpretable reasoning paths?
• RQ5 Does the use of structured knowledge graph chains reduce hallucination and improve factual consistency in LLM responses?

Key findings

• Keqing achieves a new state-of-the-art on the MetaQA benchmark with 98.4 Hits@1 on 1-hop, 99.9 on 2-hop, and 99.6 on 3-hop questions.
• Keqing-ChatGPT achieves 74.9 F1 on WebQSP, outperforming other LLM-based KBQA methods and demonstrating the superiority of the framework's workflow.
• The performance of Keqing improves with more retrieved question templates per sub-question, peaking at moderate values before diminishing due to context overload.
• Question decomposition via natural language templates outperforms direct logical chain generation (e.g., SQL-like) by a significant margin, indicating better LLM alignment with textual decomposition.
• Keqing’s use of pre-collected knowledge graph reasoning paths as CoT mentors leads to more reliable and interpretable reasoning than heuristic or end-to-end CoT methods.
• The ablation study confirms that the framework’s modular design—especially the decomposition and retrieval stages—plays a critical role in performance and robustness.

Figure 2: The pipeline of aligning decomposed sub-questions to executable logical chains on KG, where each sub-question will be mapped to a set of logical chains of top-K relevant question templates.