[Paper Review] Visual Question Answering with Prior Class Semantics

Violetta Shevchenko, Damien Teney+2 more|arXiv (Cornell University)|May 4, 2020

Multimodal Machine Learning Applications26 references4 citations

TL;DR

This paper proposes a multitask VQA framework that enhances answer prediction by incorporating prior semantic knowledge through word embeddings, using a regression objective in a semantic space to improve consistency and accuracy. The method achieves state-of-the-art performance on GQA and enables zero-shot prediction of unseen answers, demonstrating robustness across diverse question types.

ABSTRACT

We present a novel mechanism to embed prior knowledge in a model for visual question answering. The open-set nature of the task is at odds with the ubiquitous approach of training of a fixed classifier. We show how to exploit additional information pertaining to the semantics of candidate answers. We extend the answer prediction process with a regression objective in a semantic space, in which we project candidate answers using prior knowledge derived from word embeddings. We perform an extensive study of learned representations with the GQA dataset, revealing that important semantic information is captured in the relations between embeddings in the answer space. Our method brings improvements in consistency and accuracy over a range of question types. Experiments with novel answers, unseen during training, indicate the method's potential for open-set prediction.

Motivation & Objective

• To address the limitations of fixed-classification VQA models that ignore semantic relationships between answers.
• To improve model generalization and robustness by incorporating prior knowledge about answer semantics.
• To enable open-set prediction of novel, out-of-vocabulary answers not seen during training.
• To investigate how learned semantic representations of answers enhance VQA performance across diverse question types.

Proposed method

• The model treats VQA as a multitask learning problem, combining classification over candidate answers with a regression objective in a semantic embedding space.
• Answer representations are initialized using pretrained GloVe word embeddings to inject prior semantic knowledge.
• A joint loss function combines cross-entropy classification loss with a mean squared error regression loss on answer embeddings.
• At test time, the model can predict novel answers by replacing the answer matrix with new embeddings and disabling the classification loss (λ = 0).
• The method leverages semantic relations between answers, as revealed by nearest-neighbor analysis in the embedding space.
• The model is evaluated on GQA and VQA v2, with ablation studies to isolate the contribution of semantic regularization.

Experimental results

Research questions

• RQ1 How does incorporating semantic knowledge about answers improve VQA model accuracy and consistency?
• RQ2 Can a VQA model generalize to answers not seen during training using semantic embeddings?
• RQ3 What role do semantic relations between answer words play in improving model performance?
• RQ4 How does the multitask learning setup with semantic regression compare to standard classification-only baselines?
• RQ5 To what extent do different types of pretrained embeddings (e.g., GloVe vs. ConceptNet) affect zero-shot answer prediction?

Key findings

• The model achieves consistent accuracy improvements across all question categories on the GQA dataset, with the highest gains in complex and compositional questions.
• The ablation study shows that a balanced trade-off (λ = 0.5) between classification and regression losses yields the best performance, confirming the complementary nature of both objectives.
• The model improves answer recall for 70% of answer candidates, with the largest gains observed for semantically related answers (e.g., 'basket' and 'baskets').
• On the VQA v2 dataset with disjoint train/test answer sets, the model achieves 27% accuracy in zero-shot prediction, outperforming the fPMC baseline (15%).
• Answers with degraded recall are often those with semantically unrelated nearest neighbors in the GloVe space, indicating that embedding quality and semantic coherence matter.
• Visual co-occurrence patterns learned end-to-end in the VQA model suggest that combining visual and linguistic knowledge could further improve out-of-vocabulary performance.