[Paper Review] Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering

Yu Zhou, Jun Yu+2 more|arXiv (Cornell University)|Aug 4, 2017

Multimodal Machine Learning Applications36 references101 citations

TL;DR

The paper introduces Multi-modal Factorized Bilinear (MFB) pooling to fuse image and question features, plus a co-attention mechanism, achieving state-of-the-art results on VQA.

ABSTRACT

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multi-modal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multi-modal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a co-attention mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-the-art performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb.

Motivation & Objective

• Motivate improving VQA by better multi-modal feature fusion and fine-grained attention.
• Develop an efficient, expressive bilinear pooling method to fuse image and question features.
• Incorporate a co-attention mechanism to jointly learn image and question attentions.
• Demonstrate that normalization techniques are crucial for bilinear models.
• Achieve state-of-the-art results on the real-world VQA dataset using a single model.

Proposed method

• Propose Multi-modal Factorized Bilinear (MFB) pooling to fuse image and question features with low-rank factorization.
• Factorize bilinear weights into U and V matrices and apply sum pooling after element-wise multiplication.
• Apply power normalization and l2 normalization after MFB to stabilize training.
• Introduce a co-attention learning module to jointly learn image and question attentions in an end-to-end network.
• Use a ResNet-152 image encoder and a two-layer LSTM-question encoder to produce multi-modal fused features for answer classification.
• Evaluate with Open-Ended and Multiple-Choice tasks on the VQA dataset, comparing against MCB/MLB baselines and ablations.

Experimental results

Research questions

• RQ1 Does the proposed MFB pooling improve multi-modal fusion over existing bilinear pooling methods (MCB, MLB) for VQA?
• RQ2 Does incorporating power and l2 normalization improve training stability and accuracy?
• RQ3 Can co-attention learning jointly enhance image and question attentions to boost VQA performance?
• RQ4 How does the MFB-based architecture compare to state-of-the-art methods on the VQA dataset under various settings (with/without attention, with external embeddings, VG data)?

Key findings

• MFB outperforms MCB and MLB baselines in Open-Ended VQA accuracy.
• Increasing the factor k from 1 to 5 improves performance; further increases show diminishing returns, with saturation around k=10.
• Power normalization and especially l2 normalization substantially improve performance and training stability.
• Co-attention (MFB+CoAtt) yields additional accuracy gains over single-attention models.
• With external GloVe embeddings and Visual Genome data, MFB+CoAtt+GloVe+VG achieves top performance among single models on OE and MC tasks; ensembles further improve results.
• Compared with state-of-the-art, MFB+CoAtt+GloVe+VG approaches or surpasses many non-ensemble methods; human performance remains higher.