[Paper Review] Multimodal Integration of Human-Like Attention in Visual Question Answering

Ekta Sood, Fabian Kögel+4 more|arXiv (Cornell University)|Sep 27, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes MULAN, the first method to integrate human-like attention from both text and image modalities into a transformer-based VQA model. By modifying self-attention scoring functions with predictions from state-of-the-art text and image saliency models, MULAN achieves state-of-the-art accuracy of 73.98% on VQAv2 test-std and 73.72% on test-dev with approximately 80% fewer trainable parameters than prior models.

ABSTRACT

Human-like attention as a supervisory signal to guide neural attention has shown significant promise but is currently limited to uni-modal integration - even for inherently multimodal tasks such as visual question answering (VQA). We present the Multimodal Human-like Attention Network (MULAN) - the first method for multimodal integration of human-like attention on image and text during training of VQA models. MULAN integrates attention predictions from two state-of-the-art text and image saliency models into neural self-attention layers of a recent transformer-based VQA model. Through evaluations on the challenging VQAv2 dataset, we show that MULAN achieves a new state-of-the-art performance of 73.98% accuracy on test-std and 73.72% on test-dev and, at the same time, has approximately 80% fewer trainable parameters than prior work. Overall, our work underlines the potential of integrating multimodal human-like and neural attention for VQA

Motivation & Objective

• To address the limitation of prior work that only integrates human-like attention on images, not text, in visual question answering.
• To explore whether multimodal human-like attention can improve reasoning and generalization in VQA models.
• To reduce model complexity while maintaining or improving performance through inductive bias from human attention.
• To evaluate the impact of multimodal human-like attention on long and compositional questions.
• To provide interpretability by analyzing attention distributions and failure cases.

Proposed method

• MULAN integrates predictions from a text saliency model (TSM) and an image saliency model (multi-duration model) into the attention scoring functions of a MCAN-based VQA transformer.
• The method modifies the self-attention mechanism by incorporating human-like attention maps as inductive biases during the computation of query-key interactions.
• The model is trained end-to-end on the VQAv2 dataset using the MCAN small architecture as a backbone.
• Human attention on text is modeled using the recently proposed Text Saliency Model (TSM), adapted for VQA.
• Image saliency predictions are obtained from a multi-duration saliency model that captures temporal dynamics of visual attention.
• The integration is performed at the attention layer level, directly modifying the attention weights before feature fusion.

Experimental results

Research questions

• RQ1 Can multimodal human-like attention from both text and image improve VQA performance beyond unimodal supervision?
• RQ2 Does integrating human-like attention reduce reliance on dataset biases, especially in long or complex questions?
• RQ3 Can human attention maps serve as an effective inductive bias to improve model generalization and reduce parameter count?
• RQ4 How does multimodal human-like attention affect attention distributions and model interpretability?
• RQ5 Does the integration lead to more stable attention convergence during training compared to standard models?

Key findings

• MULAN achieves a new state-of-the-art accuracy of 73.98% on the VQAv2 test-std split and 73.72% on test-dev, outperforming prior methods.
• The model reduces trainable parameters by approximately 80% compared to prior SOTA models while maintaining superior performance.
• MULAN shows significant relative improvements on longer questions (7+ tokens), with relative gains of 0.3% or more, indicating better handling of complex queries.
• Attention visualization confirms that MULAN focuses on relevant image regions and key question tokens (e.g., 'digging', 'fridge'), while the baseline model spreads attention more broadly.
• The model converges faster to stable attention distributions, suggesting that initial human attention maps guide effective learning.
• In category-wise analysis, MULAN achieves the highest accuracy in 10 out of 12 categories, with notable gains in activity recognition and sentiment analysis.