[Paper Review] Retrieving Supporting Evidence for LLMs Generated Answers

Siqing Huo, Negar Arabzadeh+1 more|arXiv (Cornell University)|Jun 23, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a simple self-verification method for LLM-generated answers by retrieving supporting evidence from a corpus using a combined query of the original question and the LLM's answer. The LLM then evaluates consistency between its generated answer and the retrieved passage, achieving ~70-80% accuracy in detecting hallucinations, demonstrating promise but limited reliability for full hallucination detection.

ABSTRACT

Current large language models (LLMs) can exhibit near-human levels of performance on many natural language tasks, including open-domain question answering. Unfortunately, they also convincingly hallucinate incorrect answers, so that responses to questions must be verified against external sources before they can be accepted at face value. In this paper, we report a simple experiment to automatically verify generated answers against a corpus. After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer. We then present the LLM with the combination of the question + generated answer + retrieved answer, prompting it to indicate if the generated answer can be supported by the retrieved answer. We base our experiment on questions and passages from the MS MARCO (V1) test collection, exploring three retrieval approaches ranging from standard BM25 to a full question answering stack, including a reader based on the LLM. For a large fraction of questions, we find that an LLM is capable of verifying its generated answer if appropriate supporting material is provided. However, with an accuracy of 70-80%, this approach cannot be fully relied upon to detect hallucinations.

Motivation & Objective

• To investigate whether LLMs can self-detect hallucinations by verifying their own answers against external evidence.
• To evaluate the effectiveness of different retrieval methods—ranging from BM25 to neural QA stacks—in supporting LLM-based hallucination detection.
• To assess whether using a reader model to extract concise answers from retrieved passages improves verification accuracy.
• To determine the trade-offs between retrieval effectiveness and computational efficiency in the hallucination detection pipeline.
• To explore the limitations of the approach, including potential data leakage from training data and challenges with numerical answers.

Proposed method

• Prompt the LLM to generate an answer to a given question, instructing it to act as an expert to ensure rigor.
• Construct a combined query by concatenating the original question and the LLM-generated answer for retrieval.
• Retrieve relevant passages from the MS MARCO passage corpus using multiple retrieval methods, including BM25 and neural retrieval stacks with a reader.
• Present the LLM with the original question, its generated answer, and the retrieved passage to judge consistency.
• Use a zero-shot prompting strategy to assess whether the LLM can determine if its answer is supported by the retrieved evidence.
• Evaluate performance using human-annotated labels for hallucination detection, comparing predictions against ground truth.

Experimental results

Research questions

• RQ1 Can LLMs reliably self-detect hallucinations when provided with supporting evidence retrieved from an external corpus?
• RQ2 How does the choice of retrieval method (e.g., BM25 vs. neural reader-based retrieval) affect the accuracy of hallucination detection?
• RQ3 Does using a reader model to extract a concise answer from retrieved passages improve the LLM’s ability to verify its own response?
• RQ4 What is the impact of query composition—specifically, using a combined question+answer query—on retrieval relevance and downstream verification?
• RQ5 To what extent do model-specific behaviors, such as difficulty with numerical answers, limit the reliability of the self-verification approach?

Key findings

• The LLM achieved an overall hallucination detection accuracy of approximately 70–80%, indicating that while the method is promising, it cannot be fully relied upon to detect all hallucinations.
• Neural retrieval methods with a reader component consistently produced higher-quality answers than BM25 alone, improving verification accuracy.
• Despite improvements, the reader-based retrieval stack did not change the predicted label in about 80% of cases, suggesting limited marginal gain for high-precision applications.
• The method showed particular difficulty with numerical answers, indicating a key limitation in handling quantitative hallucinations.
• Even simple BM25 retrieval achieved acceptable performance, suggesting that computationally lightweight methods can be effective for this task.
• The approach is sensitive to potential data leakage, as the MS MARCO corpus may have been part of the LLM’s training data, potentially inflating results in a best-case scenario.