[Paper Review] Self-Critical Reasoning for Robust Visual Question Answering

Jialin Wu, Raymond J. Mooney|arXiv (Cornell University)|May 24, 2019

Multimodal Machine Learning ApplicationsComputer Science37 references91 citations

TL;DR

The paper introduces a self-critical training objective for VQA that penalizes incorrect answers for being overly sensitive to the most influential image regions, using either human or QA-derived explanations to guide object importance, achieving state-of-the-art generalization on VQA-CP datasets.

ABSTRACT

Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to generalize to test data with a significantly different question-answer (QA) distribution. To address this issue, we introduce a self-critical training objective that ensures that visual explanations of correct answers match the most influential image regions more than other competitive answer candidates. The influential regions are either determined from human visual/textual explanations or automatically from just significant words in the question and answer. We evaluate our approach on the VQA generalization task using the VQA-CP dataset, achieving a new state-of-the-art i.e., 49.5% using textual explanations and 48.5% using automatically annotated regions.

Motivation & Objective

• Motivate the problem of VQA models overfitting to language priors and poor generalization under changing QA distributions.
• Propose a self-critical training objective that suppresses incorrect answers’ sensitivity to influential image regions.
• Leverage human visual/textual explanations or QA-derived object sets to identify influential regions.
• Demonstrate improved generalization on VQA-CP with UpDn-based models, with or without human explanations.

Proposed method

• Base VQA system: Bottom-Up Top-Down (UpDn) architecture.
• Identify an influential object set I per QA pair from visual explanations (VQA-HAT), textual explanations (VQA-X), or QA-based nouns.
• Compute answer sensitivity to objects using a modified Grad-CAM (no ReLU, no feature-vector weighting).
• Strengthen the most influential object via an influence-strengthening loss L_infl to ensure the correct answer is grounded on influential regions.
• Introduce a self-critical loss L_crit that minimizes the sensitivity gap for competitive incorrect answers w.r.t. the most influential object v* (weighted by cosine distance of GloVe embeddings).
• Training regimen: pre-train with L_vqa, fine-tune with L_infl, then jointly optimize L_vqa + λ_infl L_infl + λ_crit L_crit, using a bucket of top-5 competing answers; object set size is small (6) and proposals come from explanations or QA nouns.

Experimental results

Research questions

• RQ1 Can a self-critical objective reduce sensitivity of incorrect answers to the most influential image regions in VQA?
• RQ2 Does leveraging human (visual/textual) explanations or QA-derived object sets improve generalization to changing priors (VQA-CP)?
• RQ3 How does the proposed method affect grounding of correct vs. incorrect answers to influential objects across datasets?

Key findings

• Achieves new state-of-the-art on VQA-CP with textual explanations (49.5%) and competitive results with visual explanations (49.1% HAT) and QA-derived objects (48.5%).
• Outperforms prior methods (e.g., HINT) on VQA-CP by substantial margins when using VQA-HAT or VQA-X supervision.
• False sensitivity rate drops significantly after self-critical training (from 35.5% to ~20%), showing the method reduces wrong answers’ reliance on influential regions.
• Influence-strengthening plus self-critical loss yields robust gains across a range of loss weights and object-set sizes.
• QA-based influential object proposals achieve large overlap with human-provided proposals (57.1% with VQA-X, 54.3% with VQA-HAT), indicating QA-derived sets capture meaningful regions.