[Paper Review] Relation-Aware Graph Attention Network for Visual Question Answering

Linjie Li, Zhe Gan+2 more|arXiv (Cornell University)|Mar 29, 2019

Multimodal Machine Learning ApplicationsComputer Science66 references56 citations

TL;DR

This paper introduces ReGAT, a relation-aware graph attention network that models explicit and implicit inter-object relations in images, conditioned on the question, to improve VQA performance. It is compatible as a plug-and-play module for existing VQA architectures and achieves state-of-the-art results on VQA 2.0 and VQA-CP v2.

ABSTRACT

In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects. We propose a Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations. Two types of visual object relations are explored: (i) Explicit Relations that represent geometric positions and semantic interactions between objects; and (ii) Implicit Relations that capture the hidden dynamics between image regions. Experiments demonstrate that ReGAT outperforms prior state-of-the-art approaches on both VQA 2.0 and VQA-CP v2 datasets. We further show that ReGAT is compatible to existing VQA architectures, and can be used as a generic relation encoder to boost the model performance for VQA.

Motivation & Objective

• Motivation to understand semantic dynamics and interactions among image objects for semantically rich VQA.
• Develop a relation encoder that learns both explicit (spatial/semantic) and implicit relations.
• Make relation representations adaptive to the question to guide attention and fusion.
• Demonstrate that ReGAT is compatible with existing VQA architectures and improves performance.

Proposed method

• Constructs a fully-connected implicit relation graph and optionally prunes to explicit relation graphs (spatial and semantic).
• Implements a question-adaptive graph attention mechanism to encode relations, with multi-head attention.
• Separates relation encoders into implicit, spatial, and semantic branches and combines their outputs via a weighted sum during inference.
• Uses Faster R-CNN to extract object regions and features, and a GRU-based question encoder with self-attention.
• Performs multimodal fusion with existing VQA architectures and predicts answers via an MLP classifier.
• Trains relation encoders independently and ensembles their outputs with learned weights (α, β).

Experimental results

Research questions

• RQ1 Can explicit and implicit inter-object relations improve VQA performance beyond object-level features?
• RQ2 Does a question-adaptive relation encoder yield better question-region alignment than non-adaptive methods?
• RQ3 Is ReGAT compatible as a plug-and-play module for existing VQA architectures and datasets?
• RQ4 What is the contribution of each relation type (semantic, spatial, implicit) to performance across models?

Key findings

• ReGAT yields consistent improvements across Bottom-up Top-Down (BUTD), MUTAN, and BAN baselines on VQA 2.0.
• Each relation type (semantic, spatial, implicit) improves performance when used with graph attention.
• Question-adaptive relations provide additional gains beyond static relation encoders.
• On VQA-CP v2, ReGAT achieves state-of-the-art results, with notable gains when using all relations.
• Overall, integrating ReGAT significantly outperforms baselines without requiring extra feature augmentation.