[Paper Review] GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework

Hannah Sansford, Nicholas Richardson+2 more|arXiv (Cornell University)|Jul 15, 2024

Plant-based Medicinal Research6 citations

TL;DR

GraphEval represents LLM outputs as knowledge graphs and uses NLI to detect per-triple hallucinations, improving detection accuracy and enabling explanations; it also introduces GraphCorrect for partial hallucination correction.

ABSTRACT

Methods to evaluate Large Language Model (LLM) responses and detect inconsistencies, also known as hallucinations, with respect to the provided knowledge, are becoming increasingly important for LLM applications. Current metrics fall short in their ability to provide explainable decisions, systematically check all pieces of information in the response, and are often too computationally expensive to be used in practice. We present GraphEval: a hallucination evaluation framework based on representing information in Knowledge Graph (KG) structures. Our method identifies the specific triples in the KG that are prone to hallucinations and hence provides more insight into where in the response a hallucination has occurred, if at all, than previous methods. Furthermore, using our approach in conjunction with state-of-the-art natural language inference (NLI) models leads to an improvement in balanced accuracy on various hallucination benchmarks, compared to using the raw NLI models. Lastly, we explore the use of GraphEval for hallucination correction by leveraging the structure of the KG, a method we name GraphCorrect, and demonstrate that the majority of hallucinations can indeed be rectified.

Motivation & Objective

• Address the lack of explainability in hallucination detection for LLM outputs grounded in given context.
• Propose a KG-based preprocessing step to identify inconsistent triples before standard evaluation models.
• Show that GraphEval improves balanced accuracy on benchmark datasets.
• Demonstrate the potential of GraphCorrect for correcting hallucinations while preserving original text quality.

Proposed method

• Construct a knowledge graph from LLM output via prompting strategies (CoT and in-context learning).
• For each KG triple, apply an out-of-the-box NLI model to assess factual consistency against the grounding context.
• Classify the example as inconsistent if any triple exceeds a 0.5 inconsistency probability.
• Provide explainability by returning inconsistent triples; use as a preprocessing step for existing hallucination detectors.
• Optionally extend to GraphCorrect by correcting identified hallucinations per triple and reassembling the output.

Figure 1: A visualisation of the GraphEval approach. First, the LLM output is fed into the KG construction prompt to produce the KG depicted on the right. Next, each individual triple in the KG is fed into an out-of-the-box hallucination detection method, such as an NLI model, and compared to the pr

Experimental results

Research questions

• RQ1 Can KG-based preprocessing improve the accuracy of existing hallucination detection models?
• RQ2 To what extent can per-triple grounding checks reveal the exact location of hallucinations within an LLM output?
• RQ3 Is it feasible to correct hallucinations by targeting only inconsistent KG triples (GraphCorrect) while maintaining overall text fidelity?

Key findings

• GraphEval consistently improves balanced accuracy when paired with NLI models across SummEval, QAGS-C, and QAGS-X (average improvement ~6.2 points).
• The framework identifies specific KG triples that are inconsistent with the grounding context, enhancing explainability of detected hallucinations.
• GraphCorrect outperforms a direct prompting baseline in correcting hallucinations across most benchmarks, especially for longer/structurally complex outputs.
• KG-based preprocessing reduces computational burden by enabling a single LLM pass for KG construction and using cheaper NLI checks thereafter.
• When output length is short (e.g., QAGS-X), benefits of GraphEval are diminished, highlighting its strength on longer, more complex outputs.