[Paper Review] Grounding Visual Explanations (Extended Abstract)

Lisa Anne Hendricks, Ronghang Hu+2 more|arXiv (Cornell University)|Nov 17, 2017

Multimodal Machine Learning Applications7 references3 citations

TL;DR

This paper proposes a phrase-critic model that improves visual explanation quality by grounding individual attribute phrases in both the image and the generated explanation, ensuring image relevance while maintaining linguistic fluency. By using a relative-attribute ranking loss with flipped phrases as negatives, the model significantly outperforms prior work in grounding accuracy (85% vs. 79% image-relevant attributes) and produces more accurate, interpretable explanations grounded in visual evidence.

ABSTRACT

Existing models which generate textual explanations enforce task relevance through a discriminative term loss function, but such mechanisms only weakly constrain mentioned object parts to actually be present in the image. In this paper, a new model is proposed for generating explanations by utilizing localized grounding of constituent phrases in generated explanations to ensure image relevance. Specifically, we introduce a phrase-critic model to refine (re-score/re-rank) generated candidate explanations and employ a relative-attribute inspired ranking loss using "flipped" phrases as negative examples for training. At test time, our phrase-critic model takes an image and a candidate explanation as input and outputs a score indicating how well the candidate explanation is grounded in the image.

Motivation & Objective

• To address the lack of image relevance in neural network-generated textual explanations, where models may mention attributes not present in the input image.
• To resolve the conflict between generating class-discriminative explanations and ensuring those explanations are visually grounded in the input image.
• To develop a framework that jointly optimizes for both linguistic fluency and visual grounding of explanation phrases.
• To enable automatic selection of the most image-relevant explanations through a learned scoring mechanism that evaluates phrase grounding.

Proposed method

• A sequence of candidate explanations is generated using an LSTM-based model trained with a discriminative loss to emphasize class-specific attributes.
• Each explanation is factorized into attribute phrases (e.g., 'red beak', 'black face') using a rule-based chunker.
• A pre-trained grounding model (from hu2017modeling) localizes each phrase to a visual region in the image, returning a bounding box and a localization score.
• A phrase-critic model takes each (phrase, region, score) triplet and computes a single image relevance score $ S_r $, reflecting how well the phrase is grounded in the image.
• The phrase-critic is trained with a relative-attribute ranking loss, using 'flipped' phrases (e.g., changing 'red' to 'black') as negative examples to improve discrimination.
• Final explanations are ranked using a weighted combination of the phrase-critic score $ S_r $ and a fluency score $ S_f = \log P(w_{0:T}) $, avoiding over-ranking ungrammatical sentences.

Experimental results

Research questions

• RQ1 Can a model effectively assess whether a generated textual explanation is visually grounded in the input image, beyond just linguistic fluency?
• RQ2 How can we improve the image relevance of visual explanations without compromising their linguistic quality or discriminative power?
• RQ3 To what extent does grounding individual phrases in the image improve the overall accuracy and interpretability of visual explanations?
• RQ4 Can a critic model trained on relative-attribute comparisons (e.g., 'red' vs. 'black') better identify image-irrelevant explanations than standard loss functions?

Key findings

• The phrase-critic model achieves 85% image-relevant attribute mention accuracy, significantly outperforming the baseline model (79%) in grounding attribute phrases.
• The model successfully identifies and corrects incorrect mentions—such as labeling a bird’s beak as 'orange' when it is actually 'black'—by relying on visual grounding.
• Highly ranked explanations consistently mention the most discriminative and visually accurate attributes, such as 'white eye' for the Brewer Blackbird, which is a key distinguishing feature.
• The integration of fluency scoring ($ S_f $) prevents the model from favoring grammatically incorrect but visually plausible explanations, such as those repeating 'long neck' twice.
• Visual explanations produced by the system show precise localization of small regions (e.g., 'white eye') with color-coded bounding boxes matching the corresponding phrases.
• The phrase-critic model effectively distinguishes between correct and incorrect explanations even when both mention attributes present in the image, by leveraging relative-attribute comparisons.