[Paper Review] Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Akira Fukui, Dong Huk Park+4 more|arXiv (Cornell University)|Jun 6, 2016

Multimodal Machine Learning Applications53 references394 citations

TL;DR

Introduces Multimodal Compact Bilinear pooling (MCB) to efficiently fuse visual and textual features for VQA and visual grounding, achieving state-of-the-art results on VQA datasets and improved grounding accuracy.

ABSTRACT

Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.

Motivation & Objective

• Motivate expressive multimodal fusion beyond simple concatenation or element-wise operations.
• Propose MCB to approximate outer-product interactions between image and text efficiently.
• Apply MCB to VQA with attention and to visual grounding, evaluating across multiple datasets.
• Demonstrate that MCB–based models improve over strong baselines and ablations.

Proposed method

• Define Multimodal Compact Bilinear pooling (MCB) using Count Sketch projections and FFT-based convolution to approximate outer-product interactions.
• Apply MCB to fuse image features (CNN) and question embeddings (LSTM) into a 16k-dimensional joint representation.
• Integrate soft attention over spatial features by applying MCB to language-visual pairs at each grid location and predicting attention maps.
• Extend MCB with an additional attention branch for multiple glimpses and an extra MCB for answering encoding in the multiple-choice setting.
• For visual grounding, replace concatenation in GroundeR with MCB to combine phrase and visual proposals, with L2-normalized embeddings.

Experimental results

Research questions

• RQ1 Does Multimodal Compact Bilinear pooling provide a more expressive fusion than concatenation or element-wise pooling for VQA and grounding?
• RQ2 How does MCB affect performance when integrated with attention mechanisms and multiple question-answer settings?
• RQ3 What are the effects of different dimensionalities d of the MCB feature on VQA and grounding?
• RQ4 Can MCB improve state-of-the-art results on VQA datasets and grounding benchmarks across multiple datasets?

Key findings

• MCB outperforms non-bilinear pooling baselines across VQA and grounding tasks.
• Soft attention with MCB yields the best results; attention on MCB features surpasses attention on concatenation layers.
• Using 16k-dim MCB features provides the highest accuracy in open-ended VQA settings.
• Best single model (MCB with two attentions, plus Visual Genome data and GloVe) surpasses competing approaches on VQA open-ended and multiple-choice benchmarks.
• MCB-based grounding achieves state-of-the-art accuracy on Flickr30k Entities and ReferItGame datasets.