[Paper Review] Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention

Shalini Ghosh, Giedrius Burachas+2 more|arXiv (Cornell University)|Feb 15, 2019

Multimodal Machine Learning Applications34 citations

TL;DR

The paper proposes a multi-modal approach to generate natural language explanations for VQA by leveraging scene graphs and visual attention, without additional labeled explanation data, and validates it with Visual Genome data and human judgments.

ABSTRACT

In this paper, we present a novel approach for the task of eXplainable Question Answering (XQA), i.e., generating natural language (NL) explanations for the Visual Question Answering (VQA) problem. We generate NL explanations comprising of the evidence to support the answer to a question asked to an image using two sources of information: (a) annotations of entities in an image (e.g., object labels, region descriptions, relation phrases) generated from the scene graph of the image, and (b) the attention map generated by a VQA model when answering the question. We show how combining the visual attention map with the NL representation of relevant scene graph entities, carefully selected using a language model, can give reasonable textual explanations without the need of any additional collected data (explanation captions, etc). We run our algorithms on the Visual Genome (VG) dataset and conduct internal user-studies to demonstrate the efficacy of our approach over a strong baseline. We have also released a live web demo showcasing our VQA and textual explanation generation using scene graphs and visual attention.

Motivation & Objective

• Motivate explainable question answering by generating textual justifications for VQA outputs.
• Combine scene-graph based NL representations with VQA attention maps to identify evidence for explanations.
• Avoid additional annotated explanation data by using existing scene-graph annotations and attention-guided selection.
• Demonstrate through qualitative and small-scale user studies that the approach yields reasonable explanations.

Proposed method

• Use a VQA model with image features from ResNet-152 and question features from a GloVe-embedded LSTM.
• Generate an attention heatmap over image regions during VQA to identify relevant scene elements.
• Extract NL representations from scene graph entities (objects, relations, region descriptions) and score them for relevance.
• Rank scene-graph entities by a composite score combining attention alignment, language-model relevance to QA, and description length/area terms.
• Generate slot-filler explanations from top-ranked descriptions or via descriptive graph components when region descriptions are unavailable.
• Compare region-based explanations against NL-only baselines and perform a small-scale human evaluation.

Experimental results

Research questions

• RQ1 Can NL explanations grounded in scene graphs and visual attention provide useful justifications for VQA answers?
• RQ2 Does incorporating attention-guided scene-graph evidence improve the quality of explanations compared to NL-only approaches?
• RQ3 How effective are region-description-based versus object/relation-based explanations in different VG scene graphs?
• RQ4 What is the impact of combining visual and linguistic signals on user-perceived explanation relevance?

Key findings

• The multi-modal explanations outperform the NL-only baseline across human evaluation metrics in the small internal study.
• Using attention-guided scene-graph evidence yields more relevant explanations for top-ranked QA pairs.
• Top explanation sets often include region descriptions like  tennis player hitting a ball or  woman hitting a tennis ball, illustrating concrete evidence tied to the answer.
• The paper reports favorable results in 52-55% of cases depending on the metric for multi-modal explanations versus NL-only baselines.
• A/B tests with human raters show the multi-modal approach aligned better with the VQA decision process than language-model-only explanations.