[Paper Review] Minimizing Factual Inconsistency and Hallucination in Large Language Models

I Muneeswaran, Shreya Saxena+6 more|arXiv (Cornell University)|Nov 23, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a multi-stage framework that generates, verifies, and refines rationales before producing answers, significantly reducing factual inconsistencies and hallucinations in large language models. The approach improves faithfulness by 14–25% and accuracy by 16–22% over standard RAG on drug-related queries in life sciences, with fine-tuned Llama2-7B achieving 33–42% higher accuracy than non-fine-tuned versions.

ABSTRACT

Large Language Models (LLMs) are widely used in critical fields such as healthcare, education, and finance due to their remarkable proficiency in various language-related tasks. However, LLMs are prone to generating factually incorrect responses or "hallucinations," which can lead to a loss of credibility and trust among users. To address this issue, we propose a multi-stage framework that generates the rationale first, verifies and refines incorrect ones, and uses them as supporting references to generate the answer. The generated rationale enhances the transparency of the answer and our framework provides insights into how the model arrived at this answer, by using this rationale and the references to the context. In this paper, we demonstrate its effectiveness in improving the quality of responses to drug-related inquiries in the life sciences industry. Our framework improves traditional Retrieval Augmented Generation (RAG) by enabling OpenAI GPT-3.5-turbo to be 14-25% more faithful and 16-22% more accurate on two datasets. Furthermore, fine-tuning samples based on our framework improves the accuracy of smaller open-access LLMs by 33-42% and competes with RAG on commercial models.

Motivation & Objective

• To address the critical problem of factual inconsistency and hallucination in large language models (LLMs), especially in high-stakes domains like healthcare and pharmacovigilance.
• To enhance transparency and traceability in LLM-generated responses by incorporating rationale generation and source referencing.
• To improve the reliability of LLMs in industrial applications where responses must be grounded in curated knowledge repositories and verifiable sources.
• To demonstrate that the framework outperforms standard Retrieval-Augmented Generation (RAG) in faithfulness and accuracy on biomedical question-answering tasks.
• To show that fine-tuning smaller open-source LLMs using the framework’s outputs can achieve performance comparable to RAG on commercial models.

Proposed method

• The framework generates a rationale step-by-step before producing the final answer, ensuring logical progression and factual grounding.
• It includes a rationale verifier and refiner to detect and correct incorrect or inconsistent reasoning components.
• The refined rationale is used as a supporting reference to generate the final answer, improving traceability and credibility.
• The method uses retrieval from factually curated knowledge sources (e.g., PubMed, internal databases) to ground the rationale and answer.
• The framework is evaluated using two datasets: PubmedQA (public) and AEQA (internal, expert-curated for adverse drug reactions).
• Fine-tuning is applied to smaller open-source models (e.g., Llama2-7B) using data generated by the framework to improve performance.

Experimental results

Research questions

• RQ1 Can a multi-stage reasoning framework that includes rationale verification and refinement reduce factual hallucinations in LLMs?
• RQ2 How does the proposed framework compare to standard RAG in terms of faithfulness and accuracy on biomedical question-answering tasks?
• RQ3 To what extent can fine-tuning smaller open-source LLMs using the framework’s outputs improve their performance relative to non-fine-tuned versions?
• RQ4 Does the use of structured triplets instead of full paragraphs as input reduce context length without significantly degrading performance?
• RQ5 Can the framework enhance transparency by enabling traceability of answers to specific knowledge sources?

Key findings

• The framework improves faithfulness by 14.1% and accuracy by 15.82% over RAG on the PubmedQA dataset.
• On the internal AEQA dataset, the framework improves faithfulness by 25.04% and accuracy by 21.66% compared to RAG.
• Fine-tuned Llama2-7B models trained on the framework’s outputs achieve 33–42% higher accuracy than non-fine-tuned versions.
• The RAG + FE variant with full paragraph retrieval achieves 96.85% accuracy and 83.24% faithfulness, outperforming all ablation variants.
• The rationale verifier and refiner components contribute significantly, as their removal reduces accuracy to 93.78% and faithfulness to 80.26%.
• Using triplets instead of paragraphs reduces input token length by 6.5x but results in lower faithfulness (76.96%) and accuracy (91.27%) compared to paragraph-based input.