[Paper Review] Visual Query Answering by Entity-Attribute Graph Matching and Reasoning

Peixi Xiong, Huayi Zhan+3 more|arXiv (Cornell University)|Mar 16, 2019

Multimodal Machine Learning Applications39 references4 citations

TL;DR

This paper proposes a graph-based visual question answering (VQA) method that models images and natural language queries as entity-attribute graphs (GEA and Q), respectively, and uses an inference graph (GI) to complete missing attributes. By matching the query graph against the enhanced entity-attribute graph, the method achieves interpretable reasoning and outperforms state-of-the-art models on a newly created Soccer-VQA dataset with rich annotations.

ABSTRACT

Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graph-based techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph GI , is constructed via learning over labeled data. The other two graphs, referred to as query graph Q and entity-attribute graph GEA, are generated from natural language query Qnl and image Img, that are issued from users, respectively. As GEA often does not take sufficient information to answer Q, we develop techniques to infer missing information of GEA with GI . Based on GEA and Q, we provide techniques to find matches of Q in GEA, as the answer of Qnl in Img. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.

Motivation & Objective

• To address the limitations of end-to-end neural networks in VQA by introducing a more interpretable, reasoning-capable approach.
• To model visual scenes and natural language queries as structured entity-attribute graphs for precise semantic matching.
• To enhance incomplete entity-attribute graphs using an inference graph trained on labeled data to recover missing attributes.
• To develop a graph matching mechanism that identifies relevant visual elements to answer complex visual queries.
• To create a new, richly annotated VQA dataset (Soccer-VQA) to support evaluation and future research.

Proposed method

• The method constructs an inference graph (GI) from labeled VQA data to learn common attribute relationships and infer missing attributes in the entity-attribute graph (GEA).
• It generates a query graph (Q) from natural language questions by parsing entities and attributes using dependency parsing and semantic role labeling.
• The entity-attribute graph (GEA) is built from image features using object detection and attribute prediction models, capturing visual entities and their properties.
• Missing attributes in GEA are inferred by matching subgraphs in GEA with subgraphs in GI, using a differentiable graph matching mechanism.
• A graph matching module aligns the query graph Q with the enhanced GEA to identify the answer by finding the best-matching visual elements.
• The entire framework is trained end-to-end using a differentiable relaxation of graph matching to optimize for answer accuracy.

Experimental results

Research questions

• RQ1 Can graph-based reasoning improve interpretability and performance in visual question answering compared to end-to-end neural networks?
• RQ2 How effectively can an inference graph trained on labeled data complete missing attributes in an entity-attribute graph derived from an image?
• RQ3 To what extent does graph matching between query and enhanced entity-attribute graphs improve answer accuracy on complex visual queries?
• RQ4 Can a newly created, richly annotated dataset (Soccer-VQA) support more accurate and diverse VQA evaluation?
• RQ5 How does the proposed method generalize across diverse visual and linguistic patterns in real-world scenes?

Key findings

• The proposed method outperforms the state-of-the-art VQA model on the Soccer-VQA dataset, demonstrating superior accuracy in complex visual reasoning tasks.
• The use of an inference graph significantly improves attribute completion in the entity-attribute graph, especially for rare or ambiguous attributes.
• The graph matching mechanism enables precise alignment between query semantics and visual evidence, enhancing answer relevance.
• The model achieves high interpretability by explicitly showing which visual entities and attributes contribute to the answer, unlike black-box neural networks.
• The newly created Soccer-VQA dataset provides rich, detailed annotations that support complex reasoning and improve model generalization.
• Quantitative results show a significant improvement in VQA accuracy over baseline models, particularly on compositional and attribute-based questions.