[Paper Review] A Picture May Be Worth a Hundred Words for Visual Question Answering

Yusuke Hirota, Noa García+5 more|arXiv (Cornell University)|Jun 25, 2021

Multimodal Machine Learning Applications67 references4 citations

TL;DR

This paper proposes a language-only Transformer model that uses image descriptions (around 100 words) instead of deep visual features for visual question answering (VQA), achieving competitive performance on VQA 2.0 and VQA-CP v2. It demonstrates that high-quality textual descriptions can effectively replace complex visual features, enabling simpler, more interpretable VQA systems with reduced computational cost.

ABSTRACT

How far can we go with textual representations for understanding pictures? In image understanding, it is essential to use concise but detailed image representations. Deep visual features extracted by vision models, such as Faster R-CNN, are prevailing used in multiple tasks, and especially in visual question answering (VQA). However, conventional deep visual features may struggle to convey all the details in an image as we humans do. Meanwhile, with recent language models' progress, descriptive text may be an alternative to this problem. This paper delves into the effectiveness of textual representations for image understanding in the specific context of VQA. We propose to take description-question pairs as input, instead of deep visual features, and fed them into a language-only Transformer model, simplifying the process and the computational cost. We also experiment with data augmentation techniques to increase the diversity in the training set and avoid learning statistical bias. Extensive evaluations have shown that textual representations require only about a hundred words to compete with deep visual features on both VQA 2.0 and VQA-CP v2.

Motivation & Objective

• To investigate whether textual representations of images can effectively replace deep visual features in visual question answering (VQA).
• To evaluate the performance of a language-only Transformer model using image description-question pairs as input.
• To explore data augmentation techniques to improve generalization and reduce bias in textual VQA models.
• To determine the minimal textual description length required for competitive VQA performance.
• To provide a foundation for interpretable image representation using natural language instead of visual embeddings.

Proposed method

• The model uses a RoBERTa-based language-only Transformer to process paired input of image descriptions and questions, predicting answers end-to-end.
• Image representations are derived from human-annotated descriptions from COCO and Localized Narratives datasets, not from visual features.
• Data augmentation is applied to both questions and descriptions using back-translation and synonym replacement to increase training diversity.
• The model is fine-tuned on VQA 2.0 and VQA-CP v2 datasets using standard VQA evaluation protocols.
• Performance is evaluated using both binary and non-binary accuracy metrics to ensure consistency with prior work.
• Qualitative analysis compares predictions with those of strong baselines (BAN and LXMERT) to assess reasoning capabilities and failure modes.

Experimental results

Research questions

• RQ1 How effective are textual image descriptions (around 100 words) compared to deep visual features in VQA?
• RQ2 Can a language-only Transformer model achieve competitive performance without any visual feature input?
• RQ3 What impact do data augmentation techniques have on the robustness and generalization of textual VQA models?
• RQ4 What are the limitations of language-only models in understanding fine-grained visual details not captured in descriptions?
• RQ5 How does the quality of image descriptions affect model performance and generalization?

Key findings

• The language-only model achieved competitive performance on both VQA 2.0 and VQA-CP v2, outperforming or matching state-of-the-art models based on deep visual features.
• The model required only about 100 words of description to match or exceed the performance of models using deep visual features.
• Back-translation of questions provided the most significant performance boost among data augmentation techniques.
• The model showed high consistency in predictions, with 80% of predictions matching those of strong baselines (BAN and LXMERT) on the same questions.
• The model succeeded when descriptions contained relevant actions or object counts (e.g., 'pointing to pots', 'six snowboards'), but failed on questions requiring visual details not in the description (e.g., reading text on a plane).
• Limitations were observed in understanding subtle textual relations and details not explicitly mentioned in descriptions, highlighting the dependency on description quality.