[Paper Review] Multimodal Unified Attention Networks for Vision-and-Language Interactions

Yu Zhou, Yuhao Cui+3 more|arXiv (Cornell University)|Aug 12, 2019

Multimodal Machine Learning Applications69 references33 citations

TL;DR

MUAN introduces a unified attention block that jointly models intra- and inter-modal interactions for multimodal data, stacking them into MUAN to achieve strong performance on VQA and visual grounding without task-specific tuning.

ABSTRACT

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.

Motivation & Objective

• Motivate and develop an attention mechanism that captures both intra-modal and inter-modal interactions in multimodal data.
• Propose a unified attention (UA) block that extends self-attention with gating to reweight features.
• Stack UA blocks to form a deep MUAN architecture applicable to VQA and visual grounding.
• Demonstrate state-of-the-art or competitive results on multiple VQA and visual grounding benchmarks.

Proposed method

• Introduce Gated Self-Attention (GSA) as the basic unit, using a bilinear pooling based gating to reweight Q and K before attention.
• Define a Unified Attention (UA) block that jointly models intra-modal (VV, TT) and inter-modal (VT, TV) interactions within a single framework.
• Stack multiple UA blocks to form MUAN-L for deep multimodal reasoning, enabling end-to-end training.
• For VQA, append a [ans] token to the question and predict the answer from its attended feature.
• For visual grounding, predict region scores and refine bounding boxes via mult-task losses (ranking and regression).
• Train with Adam; use pre-extracted image and question representations (GloVe+LSTM for text; CNN/object detector features for image).

Experimental results

Research questions

• RQ1 Can intra-modal interactions (within the same modality) improve multimodal reasoning when modeled together with inter-modal interactions?
• RQ2 Does a unified attention mechanism outperform sequential self-attention followed by co-attention in VQA and visual grounding tasks?
• RQ3 What is the impact of gating in the unified attention on performance and robustness?
• RQ4 How does MUAN compare with state-of-the-art methods on standard VQA and visual grounding benchmarks?

Key findings

• MUAN improves VQA performance over several baselines, achieving 70.82% test-dev and 86.77% test-std overall accuracy on VQA-v2 with bottom-up features.
• MUAN surpasses previous methods on VQA-v2, e.g., BAN, BAN+Counter, DFAF, and MCAN in reported comparisons, and achieves top performance among single-model approaches on test-std with standard features.
• On CLEVR, MUAN attains 98.7% accuracy, approaching human-level performance and outperforming several prior methods.
• Ablation shows that both self-attention and co-attention contribute to MUAN, with co-attention being particularly impactful for shallow models; gating provides a measurable improvement across configurations.
• Deeper MUAN (larger L) generally improves accuracy up to a saturation point around L=10, indicating benefits from deeper unified reasoning with dataset-dependent limits.