[Paper Review] Inverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool

Feng Liu, Tao Xiang+3 more|arXiv (Cornell University)|Mar 16, 2018

Multimodal Machine Learning Applications3 citations

TL;DR

This paper introduces Inverse Visual Question Answering (iVQA), a novel benchmark that generates questions from image-answer pairs to assess multimodal understanding. It proposes a variational iVQA model to produce diverse, contextually relevant questions and uses iVQA to diagnose VQA models by exposing their 'belief sets'—revealing widespread erroneous beliefs, including adversarial and irrelevant question-answer pairs—demonstrating that current VQA models rely on shortcuts and suffer from poor generalization despite high accuracy on standard benchmarks.

ABSTRACT

In recent years, visual question answering (VQA) has become topical. The premise of VQA's significance as a benchmark in AI, is that both the image and textual question need to be well understood and mutually grounded in order to infer the correct answer. However, current VQA models perhaps `understand' less than initially hoped, and instead master the easier task of exploiting cues given away in the question and biases in the answer distribution. In this paper we propose the inverse problem of VQA (iVQA). The iVQA task is to generate a question that corresponds to a given image and answer pair. We propose a variational iVQA model that can generate diverse, grammatically correct and content correlated questions that match the given answer. Based on this model, we show that iVQA is an interesting benchmark for visuo-linguistic understanding, and a more challenging alternative to VQA because an iVQA model needs to understand the image better to be successful. As a second contribution, we show how to use iVQA in a novel reinforcement learning framework to diagnose any existing VQA model by way of exposing its belief set: the set of question-answer pairs that the VQA model would predict true for a given image. This provides a completely new window into what VQA models `believe' about images. We show that existing VQA models have more erroneous beliefs than previously thought, revealing their intrinsic weaknesses. Suggestions are then made on how to address these weaknesses going forward.

Motivation & Objective

• To address the limitations of current VQA benchmarks, which are vulnerable to dataset biases and shortcut learning.
• To propose iVQA as a more robust, open-world benchmark that requires deeper visual-linguistic understanding than standard VQA.
• To develop a diagnostic framework using iVQA to expose the belief sets of existing VQA models, revealing their hidden misconceptions and weaknesses.
• To identify key failure modes in VQA models, such as inconsistent beliefs, irrelevant predictions, and adversarial generalization errors.
• To propose a new, challenging VQA dataset composed of adversarial examples to evaluate future models beyond standard benchmarks.

Proposed method

• A variational iVQA model is trained to generate diverse, grammatically correct, and content-correlated questions conditioned on an image and answer pair.
• The model uses a sequence-to-sequence architecture with attention mechanisms to ensure linguistic coherence and visual relevance.
• A reinforcement learning framework is employed to generate questions that maximize the likelihood of the given answer under a pre-trained VQA model, effectively probing its belief set.
• The belief set of a VQA model is defined as the set of all question-answer pairs it would predict as correct for a given image.
• Adversarial examples are collected by generating questions that the VQA model answers incorrectly despite being visually grounded.
• A new, focused benchmark dataset of 282 adversarial examples is constructed from the belief sets of four VQA models, with human-annotated correct answers for evaluation.

Experimental results

Research questions

• RQ1 Can iVQA serve as a more robust benchmark than standard VQA by reducing susceptibility to answer bias and dataset shortcuts?
• RQ2 To what extent do existing VQA models hold erroneous or inconsistent beliefs about image content, and how can these be systematically diagnosed?
• RQ3 How do the belief sets of VQA models compare to the ground-truth annotations in standard datasets, and what types of errors are most prevalent?
• RQ4 Can iVQA-generated questions expose hidden weaknesses such as irrelevant or adversarial predictions that standard evaluation misses?
• RQ5 What are the key failure modes in current VQA models, and how can future models be designed to avoid them?

Key findings

• Existing VQA models exhibit widespread erroneous beliefs, including 40% drop in accuracy on a new adversarial benchmark, indicating poor generalization despite high performance on standard datasets.
• The belief sets of VQA models contain a significant number of 'adversarial' and 'irrelevant' question-answer pairs, revealing that models often predict confidently without visual grounding.
• The iVQA model successfully generates diverse, grammatically correct, and visually relevant questions, demonstrating its ability to model complex visual-linguistic relationships.
• The analysis reveals that models suffer from 'doublethink'—holding mutually inconsistent beliefs—highlighting a need for more consistent reasoning mechanisms.
• The proposed adversarial dataset of 282 examples effectively exposes model weaknesses, with even state-of-the-art models like MCB-att achieving only 14.54% accuracy on the most challenging examples.
• Current datasets are too sparse and biased, leading to models that fail to learn robust visual representations, suggesting a need for dense, active learning-based annotation strategies.