[Paper Review] Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators

Kuang-Huei Lee, Hamid Palangi+3 more|arXiv (Cornell University)|Sep 22, 2019

Multimodal Machine Learning Applications56 references33 citations

TL;DR

The paper introduces R-SCAN and a relation-based top-down captioner that leverage neural scene graph (visual relation) features to improve image-text matching and image captioning, trained on VrR-VG rather than VG150, achieving state-of-the-art results on Flickr30K and MSCOCO single-model performance.

ABSTRACT

Grounding language to visual relations is critical to various language-and-vision applications. In this work, we tackle two fundamental language-and-vision tasks: image-text matching and image captioning, and demonstrate that neural scene graph generators can learn effective visual relation features to facilitate grounding language to visual relations and subsequently improve the two end applications. By combining relation features with the state-of-the-art models, our experiments show significant improvement on the standard Flickr30K and MSCOCO benchmarks. Our experimental results and analysis show that relation features improve downstream models' capability of capturing visual relations in end vision-and-language applications. We also demonstrate the importance of learning scene graph generators with visually relevant relations to the effectiveness of relation features.

Motivation & Objective

• Motivate grounding language to visual relations by incorporating scene graph relation features into vision-and-language models.
• Show that relation-aware features improve image-text retrieval and captioning over strong baselines.
• Demonstrate the importance of training scene graph generators on semantically rich relation datasets (VrR-VG) rather than VG150.
• Develop a relation-aware variant of SCAN (R-SCAN) for image-text matching and a relation-based top-down captioner.
• Provide analysis confirming that visually relevant relations, learned via VrR-VG, yield meaningful gains in downstream tasks.

Proposed method

• Propose R-SCAN, a relation-aware extension of SCAN that encodes both region (object/stuff) and visual relation features from a scene graph generator.
• Use a soft attention mechanism to align words with both region and relation features, with a visual feature fusion gate to adaptively combine them per word.
• Compute word-image similarity using a gated fusion of attended region and relation representations.
• Train with a hinge triplet ranking loss on hardest negatives in a mini-batch.
• Extend the top-down captioner by feeding attended relation features (from the scene graph) alongside region features into the language model.
• Pre-train a Stacked Motif Network scene graph generator on VrR-VG (avoiding Visual Genome VG150 biases) to obtain semantically rich relation embeddings.
• Optionally compare with VG150-trained relation features and demonstrate the superiority of VrR-VG-trained relations.

Experimental results

Research questions

• RQ1 Do neural scene graph learned relation features improve image-text matching compared to region-only baselines?
• RQ2 Do relation features help image captioning when integrated into a top-down captioner?
• RQ3 How does pre-training scene graph generators on VrR-VG versus VG150 affect end-task performance?
• RQ4 Is learning visually relevant relations essential for grounding language to visual content in retrieval and captioning tasks?

Key findings

• R-SCAN with VrR-VG relation features improves cross-modal retrieval over SCAN baselines on Flickr30K and MSCOCO.
• On Flickr30K 1K test, R-SCAN VrR-VG achieves recall@1 of 40.1 (text-to-image) and 39.6 (image-to-text) versus 37.9 and 38.5 for SCAN t-i AVG.
• On MSCOCO 5K test, R-SCAN VrR-VG achieves recall@1 of 57.6 (text-to-image) and 70.3 (image-to-text) versus 45.8 and 61.8 for SCAN t-i AVG in one comparison setting; Table 3 reports 36.2 (text-to-image) and 45.4 (image-to-text) for a variant.
• Pre-training relation features on VrR-VG yields better gains than VG150 for downstream tasks.
• Relation-based top-down captioning with VrR-VG relations improves CIDEr and SPICE scores over the baseline top-down captioner (CIDEr: 114.9→126.1 with cross-entropy and CIDEr optimization; SPICE: 20.9→21.8).
• The approach avoids graph convolution networks and directly leverages scene-graph-derived relation embeddings for grounding language in vision.