[Paper Review] Learning Conditioned Graph Structures for Interpretable Visual Question Answering

Will Norcliffe-Brown, Efstathios Vafeias+1 more|arXiv (Cornell University)|Jun 19, 2018

Multimodal Machine Learning Applications150 citations

TL;DR

The paper introduces a graph learner that builds a question-conditioned image graph for VQA, enabling interpretable graph convolutions and achieving 66.18% accuracy on VQA v2 test set.

ABSTRACT

Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which can capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain promising results with 66.18% accuracy and demonstrate the interpretability of the proposed method. Code can be found at github.com/aimbrain/vqa-project.

Motivation & Objective

• Motivate interpretable VQA by modeling semantic and spatial relationships via learned graphs.
• Propose a graph learner that conditions edges on the question to capture relevant object interactions.
• Integrate graph convolutions over the learned graph to produce question-aware image representations.
• Demonstrate interpretability by visualizing learned nodes and edges and compare to baselines on VQA v2.

Proposed method

• Represent image as object bounding boxes with features and spatial coordinates.
• Encode the question with word embeddings and a dynamic GRU to obtain q.
• Learn a question-conditioned adjacency matrix A via a joint embedding e from [v; q], with A = EE^T and sparsify by taking top-m neighbors.
• Perform K- kernel spatial graph convolutions using a patch operator with Gaussian kernels on pseudo-coordinates u(i,j) to produce H.
• Compute a max-pooled graph feature h_max, fuse with q via element-wise product, and classify with a 2-layer MLP.
• Train with a multi-label soft loss suitable for multiple plausible answers per question.

Experimental results

Research questions

• RQ1 Can a learned, question-conditioned graph structure improve reasoning in VQA over fixed or heuristic graph constructions?
• RQ2 Does integrating graph-based message passing over a sparse, interpretable graph lead to competitive accuracy on VQA v2 while enabling interpretability?

Key findings

• Achieves 66.18% accuracy on the VQA v2 test set, competitive with state-of-the-art baselines.
• Learning a sparse, question-specific graph improves focus on relevant objects and relationships compared to non-graph or naive graph baselines.
• Graph attention and edge weights provide interpretable visualizations of which objects and relations the model relies on.
• The model performs notably well on numeric counting-type questions relative to some baselines.
• A k-nearest neighbor graph and a simple attention baseline were outperformed by the learned, question-conditioned graph.