[Paper Review] Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering

Armin Toroghi, Willis Guo+2 more|arXiv (Cornell University)|Mar 3, 2024

Topic Modeling4 citations

TL;DR

This paper proposes $R^{3}$, a verifiable commonsense knowledge graph question answering framework that grounds all reasoning steps in knowledge graph triples and axiomatically surfaces intrinsic commonsense knowledge from large language models (LLMs). It significantly reduces hallucination and reasoning errors—especially on long-tail entities—while improving accuracy and verifiability across question answering, claim verification, and preference matching tasks compared to existing LLM-based methods.

ABSTRACT

Knowledge Graph Question Answering (KGQA) methods seek to answer Natural Language questions using the relational information stored in Knowledge Graphs (KGs). With the recent advancements of Large Language Models (LLMs) and their remarkable reasoning abilities, there is a growing trend to leverage them for KGQA. However, existing methodologies have only focused on answering factual questions, e.g., "In which city was Silvio Berlusconi's first wife born?", leaving questions involving commonsense reasoning that real-world users may pose more often, e.g., "Do I need separate visas to see the Venus of Willendorf and attend the Olympics this summer?" unaddressed. In this work, we first observe that existing LLM-based methods for KGQA struggle with hallucination on such questions, especially on queries targeting long-tail entities (e.g., non-mainstream and recent entities), thus hindering their applicability in real-world applications especially since their reasoning processes are not easily verifiable. In response, we propose Right for Right Reasons (R3), a commonsense KGQA methodology that allows for a verifiable reasoning procedure by axiomatically surfacing intrinsic commonsense knowledge of LLMs and grounding every factual reasoning step on KG triples. Through experimental evaluations across three different tasks--question answering, claim verification, and preference matching--our findings showcase R3 as a superior approach, outperforming existing methodologies and notably reducing instances of hallucination and reasoning errors.

Motivation & Objective

• To address the lack of verifiability in LLM-based knowledge graph question answering (KGQA) systems, especially for commonsense reasoning tasks.
• To reduce hallucination in LLMs when reasoning over long-tail or obscure entities in KGQA.
• To ensure that both factual reasoning steps and commonsense inferences are grounded and verifiable through explicit axiomatization.
• To improve robustness and reliability in real-world KGQA applications where high-stakes or personalized queries are common.
• To develop a methodology that maintains high performance even when entities are rare or recently introduced, overcoming limitations of existing LLM-based baselines.

Proposed method

• A tree-structured search is used to enforce grounding of every factual reasoning step on relevant KG triples, enabling traceability and verification.
• Intrinsic commonsense knowledge from LLMs is axiomatically surfaced to guide reasoning without relying on external knowledge sources.
• The framework explicitly separates factual reasoning (grounded in KG triples) from commonsense inference (guided by surfaced axioms), ensuring verifiability.
• A conservative, rigorous prompting strategy is employed to avoid hallucination by requiring all factual premises to be supported by KG triples.
• The method uses few-shot demonstrations and task-specific prompts to guide the LLM in generating structured, verifiable reasoning chains.
• The system maintains a reasoning budget and terminates early if no valid path is found, prioritizing correctness over completeness.

Experimental results

Research questions

• RQ1 Can LLM-based KGQA systems be made verifiable for commonsense reasoning tasks by grounding all factual steps in KG triples?
• RQ2 How does the performance of LLM-based KGQA degrade on long-tail entities, and can this be mitigated through verifiable reasoning?
• RQ3 To what extent does axiomatically surfacing intrinsic commonsense knowledge reduce hallucination and improve reasoning reliability?
• RQ4 How does $R^{3}$ compare to existing LLM-based and pure LLM baselines in terms of accuracy, FActScore, and reasoning quality across diverse KGQA tasks?
• RQ5 Can $R^{3}$ maintain high performance in personalized KGQA tasks such as preference matching, where entity familiarity is limited?

Key findings

• $R^{3}$ achieves the highest accuracy (57%) in the preference matching task, significantly outperforming KAPING (44%) and demonstrating superior personalization capabilities.
• In claim verification, $R^{3}$ maintains a high FActScore of 0.98 in both original and long-tail settings, while baselines show significant drops in FActScore under long-tail conditions.
• On the long-tail setting, $R^{3}$ maintains 98% of its original accuracy, whereas 2-shot CoT drops by 0.51 points, highlighting its robustness to entity obscurity.
• $R^{3}$ achieves the highest reasoning score (0.04) among all methods, outperforming few-shot CoT and demonstrating more reliable reasoning chains.
• The method reduces hallucination and reasoning errors by grounding all factual steps in KG triples and explicitly surfacing commonsense axioms, leading to more trustworthy outputs.
• Despite its conservative approach, $R^{3}$ achieves strong performance with minimal hallucination, indicating that verifiability and correctness can be prioritized without sacrificing accuracy.