[Paper Review] Bilinear Attention Networks

Jin-Hwa Kim, Jae-Hyun Jun+1 more|arXiv (Cornell University)|May 21, 2018

Multimodal Machine Learning Applications28 references78 citations

TL;DR

BAN learns bilinear attention maps over multi-channel vision-language inputs using low-rank pooling and residual attention, achieving state-of-the-art results on VQA 2.0 and Flickr30k Entities.

ABSTRACT

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

Motivation & Objective

• Motivate improved fusion of vision and language beyond co-attention by modeling interactions between modality channels.
• Propose a bilinear attention mechanism that jointly attends over two groups of input channels.
• Introduce a residual learning scheme to efficiently utilize multiple bilinear attention maps.
• Evaluate BAN on VQA 2.0 and Flickr30k Entities to establish state-of-the-art performance and grounding capability.

Proposed method

• Define a bilinear attention map A between two multi-channel inputs X and Y and compute joint representations via low-rank bilinear pooling.
• Parameterize A with softmax over a bilinear score that uses Hadamard products and low-rank projections (U, V, p).
• Extend to multiple glimpses by learning A_g with shared U, V and distinct p_g.
• Apply a multimodal residual network variant to integrate multiple BAN maps without concatenation, enabling 8-glimpse learning.
• Use ReLU nonlinearities in both feature interactions and attentions, with a two-layer MLP classifier for VQA and BCE losses for grounding tasks.

Experimental results

Research questions

• RQ1 Can bilinear attention capture interactions between vision and language channels more effectively than co-attention or unitary attention?
• RQ2 Does residual integration of multiple bilinear attention maps improve accuracy and efficiency?
• RQ3 How does BAN perform on VQA 2.0 and Flickr30k Entities in terms of accuracy and grounding speed?
• RQ4 What is the impact of the number of glimpses on performance and robustness?

Key findings

• BAN with bilinear attention maps outperforms unitary attention and co-attention on VQA 2.0 validation.
• Increasing glimpses improves VQA validation scores (BAN-1: 65.36, BAN-2: 65.61, BAN-4: 65.81, BAN-8: 66.00, BAN-12: 66.04).
• Residual learning of attention yields better results than sum or concat fusion of multiple BAN maps.
• On Flickr30k Entities, BAN achieves 69.69% Recall@1, surpassing prior methods without extra features, and inference speed improves by 25.37% (0.67 ms/entity).
• BAN demonstrates competitive visual grounding and maintains parameter efficiency with eight glimpses.
• The model attains state-of-the-art results on both VQA 2.0 and Flickr30k Entities benchmarks.