[Paper Review] Overcoming Language Priors in Visual Question Answering with Adversarial Regularization

Sainandan Ramakrishnan, Aishwarya Agrawal+1 more|arXiv (Cornell University)|Oct 8, 2018

Multimodal Machine Learning ApplicationsComputer Science104 citations

TL;DR

This paper introduces a question-only adversary and a difference-of-entropies regularizer to reduce language priors in VQA, improving bias-sensitive performance on VQA-CP while preserving standard VQA accuracy.

ABSTRACT

Modern Visual Question Answering (VQA) models have been shown to rely heavily on superficial correlations between question and answer words learned during training such as overwhelmingly reporting the type of room as kitchen or the sport being played as tennis, irrespective of the image. Most alarmingly, this shortcoming is often not well reflected during evaluation because the same strong priors exist in test distributions; however, a VQA system that fails to ground questions in image content would likely perform poorly in real-world settings. In this work, we present a novel regularization scheme for VQA that reduces this effect. We introduce a question-only model that takes as input the question encoding from the VQA model and must leverage language biases in order to succeed. We then pose training as an adversarial game between the VQA model and this question-only adversary -- discouraging the VQA model from capturing language biases in its question encoding. Further,we leverage this question-only model to estimate the increase in model confidence after considering the image, which we maximize explicitly to encourage visual grounding. Our approach is a model agnostic training procedure and simple to implement. We show empirically that it can improve performance significantly on a bias-sensitive split of the VQA dataset for multiple base models -- achieving state-of-the-art on this task. Further, on standard VQA tasks, our approach shows significantly less drop in accuracy compared to existing bias-reducing VQA models.

Motivation & Objective

• Motivate and quantify the problem of language priors in Visual Question Answering (VQA) and their impact on grounding.
• Propose a training-time regularization that minimizes question-only discriminative power.
• Introduce a second regularizer that maximizes the information gain from the image to the answer.
• Demonstrate improvements on bias-sensitive splits (VQA-CP) and robustness on standard VQA benchmarks.
• Show that the method is model-agnostic and can be applied on top of existing VQA architectures.

Proposed method

• Define a base VQA model with image and question encoders (f, g, h).
• Introduce a question-only adversary f_Q that predicts answers using only the question encoding q = g(Q).
• Train with an adversarial objective that minimizes VQA loss while maximizing the question-only loss (gradient reversal on q).
• Add a difference-of-entropies regularizer L_H to encourage the image to provide reducing uncertainty about the answer (maximize H(A|Q) - H(A|I,Q)).
• Joint objective: min_f,g,h max_f_Q L_VQA - lambda_Q L_QA - lambda_H L_H, where lambda_Q and lambda_H control regularization strength.
• Explain implementation details: a small 2-layer MLP for f_Q; end-to-end training; use of DoE to stabilize training.

Experimental results

Research questions

• RQ1 Can a question-only adversary reduce the reliance of VQA models on language priors?
• RQ2 Does adversarial regularization improve grounding without sacrificing standard VQA performance?
• RQ3 Does incorporating a difference-of-entropies term further enhance visual grounding and bias mitigation?
• RQ4 Are the proposed regularizers model-agnostic and compatible with existing VQA architectures?
• RQ5 How do results vary across bias-sensitive (VQA-CP) and standard (VQA-v2) benchmarks?

Key findings

• Significant improvements on VQA-CP v2 across base models (SAN and UpDn) when adding Q-Adv, DoE, or both.
• SAN + Q-Adv + DoE achieves the best bias-robust performance, surpassing previous state-of-the-art GVQA on VQA-CP v2.
• DoE and Q-Adv provide complementary gains, with their combination yielding the largest improvements.
• On standard VQA benchmarks, the proposed regularizers cause only small drops in accuracy compared to larger drops for biased baselines like GVQA.
• Question-only regularization reduces the question-only model’s accuracy on VQA-CP v2 training (indicating reduced language discriminability in the question embedding).
• The approach yields state-of-the-art results on VQA-CP and demonstrates more stable distributions and attention heatmaps, indicating better grounding.