[Paper Review] Mitigating Knowledge Conflicts in Language Model-Driven Question Answering

Han Cao, Zhaoyang Zhang+4 more|arXiv (Cornell University)|Nov 18, 2024

Topic Modeling4 citations

TL;DR

This paper proposes prompt tuning with bottleneck and prefix adapters to mitigate hallucination in language model-driven question answering by reducing over-reliance on memorized parametric knowledge. The methods successfully override incorrect memorizations, achieving 92.9% accuracy on the KMIR test set and 64.8% on NQ, demonstrating improved fidelity to input context without performance sacrifice.

ABSTRACT

In the context of knowledge-driven seq-to-seq generation tasks, such as document-based question answering and document summarization systems, two fundamental knowledge sources play crucial roles: the inherent knowledge embedded within model parameters and the external knowledge obtained through context. Recent studies revealed a significant challenge: when there exists a misalignment between the model's inherent knowledge and the ground truth answers in training data, the system may exhibit problematic behaviors during inference, such as ignoring input context, or generating unfaithful content. Our investigation proposes a strategy to minimize hallucination by building explicit connection between source inputs and generated outputs. We specifically target a common hallucination pattern in question answering, examining how the correspondence between entities and their contexts during model training influences the system's performance at inference time.

Motivation & Objective

• To address factual hallucination in language model-driven question answering caused by over-reliance on parametric knowledge.
• To mitigate entity-based knowledge conflicts where models ignore input context and instead generate answers based on memorized facts.
• To develop training schemes that constrain model behavior under insufficient grounding context, improving faithfulness to input.
• To evaluate whether prompt tuning with adapters can override memorized knowledge without sacrificing performance.
• To demonstrate that explicit correlation between input context and generated output reduces hallucination.

Proposed method

• Proposes two orthogonal prompt tuning methods: bottleneck adapter and prefix tuning adapter to control model behavior.
• Uses shuffled, contextually correct answers as new targets during fine-tuning to override memorized incorrect answers.
• Applies gradient-based decoding to encourage attention to input context during inference.
• Employs adapter-based fine-tuning to explicitly align generated answers with input context, minimizing reliance on parametric knowledge.
• Trains models on datasets with substituted answers to test whether they can be steered toward correct, contextually grounded outputs.
• Evaluates performance using exact match accuracy on original and modified examples to measure attention to input changes.

Experimental results

Research questions

• RQ1 Can prompt tuning methods override memorized parametric knowledge in language models to reduce hallucination?
• RQ2 To what extent can bottleneck and prefix adapters improve model fidelity to input context in question answering?
• RQ3 Does explicit correlation between input context and generated output reduce entity-based knowledge conflicts?
• RQ4 How does the model's performance change when the correct answer is substituted with a contextually accurate one?
• RQ5 Can the model be trained to consistently generate answers grounded in input context rather than memorized facts?

Key findings

• The bottleneck adapter achieved 92.9% accuracy on the KMIR test set, demonstrating strong performance in overriding memorized answers.
• On the more complex NQ dataset, the model achieved 64.8% accuracy using prefix tuning, indicating effective attention to context despite complexity.
• Both methods successfully changed the model’s output from memorized incorrect answers to contextually correct ones, as shown in qualitative examples.
• The model’s ability to generate the substituted answer was confirmed through exact match evaluation, showing improved attention to input context.
• The results indicate that prompt tuning with adapters can effectively mitigate hallucination without significant performance degradation.
• The study shows that explicit correlation between input and output reduces over-reliance on parametric knowledge, enhancing factual consistency.