[Paper Review] RUBi: Reducing Unimodal Biases in Visual Question Answering

Rémi Cadène, Corentin Dancette+3 more|arXiv (Cornell University)|Jun 24, 2019

Multimodal Machine Learning Applications48 references205 citations

TL;DR

RUBi introduces a question-only branch during VQA training to downweight unimodal biases, improving robustness on biased datasets like VQA-CP v2.

ABSTRACT

Visual Question Answering (VQA) is the task of answering questions about an image. Some VQA models often exploit unimodal biases to provide the correct answer without using the image information. As a result, they suffer from a huge drop in performance when evaluated on data outside their training set distribution. This critical issue makes them unsuitable for real-world settings. We propose RUBi, a new learning strategy to reduce biases in any VQA model. It reduces the importance of the most biased examples, i.e. examples that can be correctly classified without looking at the image. It implicitly forces the VQA model to use the two input modalities instead of relying on statistical regularities between the question and the answer. We leverage a question-only model that captures the language biases by identifying when these unwanted regularities are used. It prevents the base VQA model from learning them by influencing its predictions. This leads to dynamically adjusting the loss in order to compensate for biases. We validate our contributions by surpassing the current state-of-the-art results on VQA-CP v2. This dataset is specifically designed to assess the robustness of VQA models when exposed to different question biases at test time than what was seen during training. Our code is available: github.com/cdancette/rubi.bootstrap.pytorch

Motivation & Objective

• Motivate the need to reduce question-modality biases that allow VQA models to ignore the image.
• Propose a learning strategy (RUBi) that downweights biased examples during training.
• Show that RUBi is model-agnostic and can improve performance across architectures.
• Demonstrate robustness gains on biased benchmarks while maintaining competitive VQA-v2 results.

Proposed method

• Attach a question-only branch to a base VQA model during training to capture language biases.
• Compute a mask from the question-only branch and apply it multiplicatively to the VQA outputs before loss computation.
• Jointly optimize the base VQA model parameters and the question-only branch with two losses: L_QM (main) and L_QO (question-only).
• Remove the question-only branch after training; use the base VQA model for inference.
• Demonstrate compatibility with architectures like SAN and UpDn and report improvements on VQA-CP v2.

Experimental results

Research questions

• RQ1 How much can question-only biases be captured and mitigated during training to improve VQA robustness?
• RQ2 Does RUBi improve performance across different VQA architectures on biased vs. standard datasets?
• RQ3 What is the effect of the proposed masking strategy on learning dynamics and bias reduction?
• RQ4 To what extent does reducing unimodal biases impact standard VQA-v2 performance?

Key findings

• RUBi achieves an average overall accuracy of 47.11% on VQA-CP v2, a +5.94 point improvement over the previous state-of-the-art.
• RUBi provides gains across architectures: SAN (+11.73) and UpDn (+4.5) over their baselines with bias-reduction methods.
• On VQA-CP v2, RUBi outperforms the baseline by +8.65 points, and surpasses prior bias-focused methods like GVQA by large margins.
• RUBi maintains competitive performance on VQA-v2 while achieving large gains on VQA-CP v2, indicating robustness to question biases without severely hurting standard accuracy.
• Ablation confirms the necessity of L_QO and the masking approach for bias reduction.