[Paper Review] Exploring Visual Relationship for Image Captioning

Ting Yao, Yingwei Pan+2 more|arXiv (Cornell University)|Sep 19, 2018

Multimodal Machine Learning Applications37 references49 citations

TL;DR

Introduces GCN-LSTM, a graph convolutional network-based encoder that leverages semantic and spatial relationships among detected objects to improve image captioning; achieves state-of-the-art CIDEr-D on COCO.

ABSTRACT

It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1% to 128.7% on COCO testing set.

Motivation & Objective

• Motivate the use of object relationships for richer image understanding in captioning.
• Propose a relation-aware image encoder integrating semantic and spatial graphs.
• Demonstrate improved captioning performance on COCO with graph-based attention decoding.

Proposed method

• Detect objects with Faster R-CNN to form region set V.
• Construct semantic and spatial graphs over detected regions with directional edges and labels.
• Refine region features via a labeled, directed GCN with edge-wise gates.
• Use two attention-based LSTM decoders (one per graph) to generate captions.
• Fuse outputs with late fusion by linearly combining word probabilities from both decoders.

Experimental results

Research questions

• RQ1 Do semantic and spatial relationships between objects improve image captioning beyond region-level attention alone?
• RQ2 Can a GCN over relation graphs produce more informative region representations for caption generation?
• RQ3 What is the impact of fusing semantic and spatial relation signals on caption quality?

Key findings

• GCN-LSTM variants outperform baselines (LSTM, Up-Down, SCST, ADP-ATT) on COCO across multiple metrics.
• With CIDEr-D optimization, GCN-LSTM achieves 128.7 CIDEr-D and 22.1 SPICE, surpassing the best prior method by notable margins.
• Using both semantic and spatial graphs in a late fusion yields further gains over single-graph variants.
• On COCO online testing, GCN-LSTM achieves top performance across c5 and c40 references.