[Paper Review] IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding

Lanyun Zhu, Deyi Ji+4 more|arXiv (Cornell University)|Feb 28, 2024

Hallucinations in medical conditions5 citations

TL;DR

This paper introduces Image-Biased Decoding (IBD), a contrastive decoding method that reduces LVLM hallucinations by comparing an image-biased predictor with the standard model, improving truthfulness with minimal parameter overhead.

ABSTRACT

Despite achieving rapid developments and with widespread applications, Large Vision-Language Models (LVLMs) confront a serious challenge of being prone to generating hallucinations. An over-reliance on linguistic priors has been identified as a key factor leading to these hallucinations. In this paper, we propose to alleviate this problem by introducing a novel image-biased decoding (IBD) technique. Our method derives the next-token probability distribution by contrasting predictions from a conventional LVLM with those of an image-biased LVLM, thereby amplifying the correct information highly correlated with image content while mitigating the hallucinatory errors caused by excessive dependence on text. We further conduct a comprehensive statistical analysis to validate the reliability of our method, and design an adaptive adjustment strategy to achieve robust and flexible handling under varying conditions. Experimental results across multiple evaluation metrics verify that our method, despite not requiring additional training data and only with a minimal increase in model parameters, can significantly reduce hallucinations in LVLMs and enhance the truthfulness of the generated response.

Motivation & Objective

• Motivate and address hallucinations in LVLMs caused by over-reliance on linguistic priors.
• Develop a decoding-time mechanism that emphasizes image content without retraining the model.
• Introduce a contrastive decoding framework that pairs an image-biased predictor with the standard model to improve token selection.
• Provide a statistical and empirical validation demonstrating reduced hallucinations across LVLMs with minimal overhead.

Proposed method

• Construct an image-biased LVLM hatTheta by amplifying attention on image tokens via a bias coefficient c in the QK attention computation without changing model parameters.
• Compute next-token logits by contrasting predictions from the original model theta and the image-biased model hatTheta using L_CD = logit_hatTheta - logit_theta, and derive p(y_i | y_<i) from softmax(L_CD).
• Hypothesize that the token with the highest probability boost from theta to hatTheta is more likely to be correct for content words (not function words).
• Introduce a dynamic adjustment mechanism that blends standard MLE decoding with CD-based adjustments, controlled by an adaptive factor I and a scaling parameter alpha, to handle varying word types and similarity between theta and hatTheta.
• Enhance IBD with (a) finetuning hatTheta on COCO captions via lightweight prompt tuning (P) and (b) an adaptive plausibility constraint that limits candidates to high-probability tokens.
• Present full method with equations for L_CD, the dynamic weighting, and the candidate-head constraint V_head(y_<i).

Experimental results

Research questions

• RQ1 Does image-biased decoding reduce hallucinations in LVLMs compared with baseline decoding strategies?
• RQ2 Can a contrastive decoding score derived from an image-focused predictor improve token selection for content words while avoiding function-word overcorrections?
• RQ3 What are the effects of dynamic adjustment and prompts on the robustness of IBD across LVLMs and datasets?
• RQ4 How does the image-bias interact with model-world knowledge, and are there limits (e.g., benign text-biased hallucinations) to the approach?

Key findings

• IBD consistently outperforms baseline decoding and several hallucination-focused decoders across CHAIR, GPT-4 assisted, and GPT-4V assisted evaluations.
• IBD achieves lower CHAIR scores (fewer object hallucinations) across InstructBLIP, MiniGPT-4, LLaVA-1.5, and Shikra baselines.
• With minimal parameter overhead (~74K extra prompt parameters), IBD reduces hallucinations and improves truthfulness compared with ReCaption, Woodpecker, CD, VCD, DoLa, and OPERA.
• A dynamic adjustment mechanism improves robustness by favoring MLE for function words and leveraging CD for content words or highly image-dependent predictions.
• Ablation studies show the necessity of dynamic adjustment, prompt fine-tuning, and adaptive plausibility constraints for preserving performance.
• IBD achieves top or near-top scores across CHAIR, GPT-4 assisted, and GPT-4V assisted evaluations on multiple LVLM backbones.