[Paper Review] Detecting and Preventing Hallucinations in Large Vision Language Models

Anisha Gunjal, Jihan Yin+1 more|arXiv (Cornell University)|Aug 11, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces M-HalDetect, a fine-grained multimodal hallucination detection dataset with 16k sub-sentence annotations for visual question answering. It proposes Fine-grained Direct Preference Optimization (FDPO) and reward modeling with best-of-n rejection sampling, reducing hallucination rates in InstructBLIP by 41% and 55%, respectively, and generalizing to LLaVA and mPLUG-OWL with 15% and 57% reductions.

ABSTRACT

Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.

Motivation & Objective

• To address the persistent issue of hallucinations in instruction-tuned Large Vision Language Models (LVLMs), particularly in detailed image descriptions.
• To create a comprehensive, fine-grained dataset for annotating hallucinations beyond just object-level errors, including unfaithful descriptions and relationships.
• To develop and evaluate methods—FDPO and reward modeling—for reducing hallucinations in LVLMs using the new dataset.
• To demonstrate generalization of the reward model across multiple LVLMs, including LLaVA and mPLUG-OWL.
• To establish a strong correlation between automated reward model scores and human-annotated hallucination accuracy.

Proposed method

• Constructs M-HalDetect, a dataset of 16k fine-grained annotations on VQA responses from InstructBLIP, categorizing sub-sentences as Accurate, Inaccurate, or Analysis.
• Develops Fine-grained Direct Preference Optimization (FDPO), a variant of DPO that uses per-sub-sentence annotations instead of pairwise preference comparisons.
• Trains multi-modal reward models at sentence and sub-sentence levels using M-HalDetect to evaluate hallucination likelihood.
• Applies best-of-n rejection sampling with the trained reward models to select more accurate generations from InstructBLIP.
• Evaluates FDPO and rejection sampling via human evaluation and compares performance across multiple LVLMs, including InstructBLIP, LLaVA, and mPLUG-OWL.
• Measures correlation between reward model scores and human-annotated hallucination accuracy to validate the model's reliability as an evaluator.

Figure 1: Example Annotation from the M-HalDetect Dataset. The sub-sentences of text generated by multi-modal LM are tagged into categories: Accurate, Inaccurate , and Analysis .

Experimental results

Research questions

• RQ1 Can a fine-grained, sub-sentence level hallucination detection dataset improve the detection and mitigation of hallucinations in LVLMs?
• RQ2 Does Fine-grained Direct Preference Optimization (FDPO) outperform standard DPO in reducing hallucinations when using detailed annotations?
• RQ3 To what extent can reward models trained on M-HalDetect reduce hallucinations in other LVLMs beyond InstructBLIP?
• RQ4 How well do automated reward model scores correlate with human-annotated hallucination accuracy?
• RQ5 Does rejection sampling with the reward model generalize across different LVLM architectures and reduce hallucination rates effectively?

Key findings

• FDPO reduces hallucination rates in InstructBLIP by 41% compared to the baseline, even when sampling at high temperature.
• Best-of-64 rejection sampling with the reward model reduces hallucination rates in InstructBLIP by 55%.
• The same reward model reduces hallucination rates in LLaVA by 15% and in mPLUG-OWL by 57% using best-of-16 sampling.
• The reward model scores show a strong correlation with human-annotated accuracy scores, indicating its effectiveness as an automated evaluator.
• FDPO outperforms high-temperature sampling from the original InstructBLIP, suggesting that fine-grained supervision improves generation faithfulness.
• The reward model generalizes effectively across different LVLMs, demonstrating transferability beyond the model it was trained on.

Figure 3: Confusion Matrix comparison between Binary and Ternary Classifiers. The right plot represents the binary classifier labels derived from the ternary classifier by merging the Accurate and Analysis classes.