[Paper Review] Expressing Visual Relationships via Language

Hao Tan, Franck Dernoncourt+3 more|arXiv (Cornell University)|Jun 18, 2019

Multimodal Machine Learning Applications44 references4 citations

TL;DR

This paper introduces a novel language-guided image editing dataset and proposes a relational speaker model with static and dynamic relational attention mechanisms to generate natural language descriptions of visual relationships between two images. The model outperforms all baselines on three datasets, achieving state-of-the-art results in two-image captioning through improved modeling of visual transformations and relationships.

ABSTRACT

Describing images with text is a fundamental problem in vision-language research. Current studies in this domain mostly focus on single image captioning. However, in various real applications (e.g., image editing, difference interpretation, and retrieval), generating relational captions for two images, can also be very useful. This important problem has not been explored mostly due to lack of datasets and effective models. To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions. We then propose a new relational speaker model based on an encoder-decoder architecture with static relational attention and sequential multi-head attention. We also extend the model with dynamic relational attention, which calculates visual alignment while decoding. Our models are evaluated on our newly collected and two public datasets consisting of image pairs annotated with relationship sentences. Experimental results, based on both automatic and human evaluation, demonstrate that our model outperforms all baselines and existing methods on all the datasets.

Motivation & Objective

• To address the under-explored task of generating natural language captions describing visual relationships between two images.
• To create a high-quality, diverse dataset of real image pairs with human-annotated editing instructions for language-guided image editing.
• To design a neural relational-speaker model that effectively captures complex visual relationships beyond simple differences.
• To evaluate the model on multiple datasets, including a newly collected dataset and two public benchmarks, to demonstrate generalization and robustness.
• To provide a publicly available dataset and codebase to advance research in relational visual captioning.

Proposed method

• Proposes a novel encoder-decoder architecture enhanced with static relational attention to model relationships between two images using pre-computed visual features.
• Introduces sequential multi-head attention to capture long-range dependencies and sequential reasoning in the caption generation process.
• Develops a dynamic relational attention module that computes visual alignment and relationship modeling jointly during decoding, mathematically equivalent to attending over all visual relationships.
• Trains the model using reinforcement learning with self-critical sequence training to optimize for automatic evaluation metrics like BLEU, ROUGE, and CIDEr.
• Adapts the model for zero-shot transfer by fine-tuning on diverse datasets, including NLVR2 and Spot-the-Diff, after pre-training on the new image editing dataset.
• Uses object detection features from pre-trained models as visual inputs, though acknowledges limitations in detecting small or subtle objects in certain datasets.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively generate natural language descriptions that capture complex visual relationships between two real-world images?
• RQ2 How does incorporating dynamic relational attention improve the modeling of visual transformations and relationships compared to static attention or baseline models?
• RQ3 To what extent can a model pre-trained on a large-scale image editing instruction dataset generalize to other two-image captioning benchmarks like Spot-the-Diff and NLVR2?
• RQ4 What types of visual relationships (e.g., object-level changes, spatial shifts, color adjustments) are most effectively captured by the proposed model?
• RQ5 How does the model perform in comparison to existing methods in both automatic and human evaluation settings?

Key findings

• The proposed relational speaker model with dynamic relational attention achieves state-of-the-art performance on all three evaluated datasets: the newly collected Image Editing Request dataset, Spot-the-Diff, and NLVR2.
• On the Image Editing Request dataset, the model significantly outperforms all baselines in both automatic metrics (e.g., CIDEr, BLEU) and human evaluation, demonstrating superior alignment with human-annotated instructions.
• On the Spot-the-Diff dataset, the model achieves the highest CIDEr score among all methods, with human evaluation confirming improved accuracy in describing differences, though errors still occur due to misidentified entities or missed subtle changes.
• On the NLVR2 dataset, the model shows strong performance despite the complexity of the visual reasoning task, indicating its ability to generalize to diverse visual relationship types.
• The model’s performance is sensitive to image content, particularly in detecting small or low-visibility objects, suggesting limitations in visual feature extraction when object detectors fail.
• The dynamic relational attention mechanism enables the model to implicitly learn pixel-level differences even without explicit localization, demonstrating its effectiveness in modeling visual transformations.