[Paper Review] Hierarchy Parsing for Image Captioning

Ting Yao, Yingwei Pan+2 more|arXiv (Cornell University)|Sep 9, 2019

Multimodal Machine Learning Applications37 references4 citations

TL;DR

This paper proposes HIerarchy Parsing (HIP), a novel image encoder architecture that models visual understanding through a three-level hierarchy—instances, regions, and the whole image—using a Tree-LSTM to aggregate features hierarchically. HIP boosts image captioning performance, achieving 130.6 CIDEr-D on COCO Karpathy test split when combined with GCN-enhanced features, setting a new state-of-the-art.

ABSTRACT

It is always well believed that parsing an image into constituent visual patterns would be helpful for understanding and representing an image. Nevertheless, there has not been evidence in support of the idea on describing an image with a natural-language utterance. In this paper, we introduce a new design to model a hierarchy from instance level (segmentation), region level (detection) to the whole image to delve into a thorough image understanding for captioning. Specifically, we present a HIerarchy Parsing (HIP) architecture that novelly integrates hierarchical structure into image encoder. Technically, an image decomposes into a set of regions and some of the regions are resolved into finer ones. Each region then regresses to an instance, i.e., foreground of the region. Such process naturally builds a hierarchal tree. A tree-structured Long Short-Term Memory (Tree-LSTM) network is then employed to interpret the hierarchal structure and enhance all the instance-level, region-level and image-level features. Our HIP is appealing in view that it is pluggable to any neural captioning models. Extensive experiments on COCO image captioning dataset demonstrate the superiority of HIP. More remarkably, HIP plus a top-down attention-based LSTM decoder increases CIDEr-D performance from 120.1% to 127.2% on COCO Karpathy test split. When further endowing instance-level and region-level features from HIP with semantic relation learnt through Graph Convolutional Networks (GCN), CIDEr-D is boosted up to 130.6%.

Motivation & Objective

• To improve image captioning by modeling hierarchical visual structure from instance to image level.
• To address the lack of compositional visual understanding in existing neural captioning models.
• To design a flexible, plug-in feature refiner that enhances multi-level visual representations for captioning.
• To validate the generalizability of the hierarchy-aware feature learning on recognition tasks.

Proposed method

• The method uses Faster R-CNN and Mask R-CNN to detect and segment regions and instances, forming a three-level hierarchy.
• A tree-structured Long Short-Term Memory (Tree-LSTM) network processes features from the leaf level (instances) up to the root (whole image), aggregating hierarchical context.
• Instance and region features are enhanced by constructing a semantic graph with directed edges between visual elements, modeling visual relations.
• Graph Convolutional Networks (GCN) are applied on the semantic graph to enrich instance and region features with relational context.
• The hybrid of image-level, region-level, and instance-level features from HIP is fed into a standard attention-based LSTM decoder for sentence generation.
• The framework is designed to be plug-and-play, compatible with existing captioning models like Up-Down and GCN-LSTM.

Experimental results

Research questions

• RQ1 Can modeling a hierarchical structure from instances to regions to the full image improve image captioning performance?
• RQ2 How does Tree-LSTM-based aggregation of hierarchical features enhance visual representation for captioning?
• RQ3 To what extent does incorporating visual relations via GCN improve captioning quality beyond hierarchical feature fusion?
• RQ4 Can the proposed hierarchy parsing framework generalize beyond captioning to other vision tasks like recognition?
• RQ5 What is the optimal threshold for constructing the hierarchical structure in terms of performance?

Key findings

• HIP with a top-down attention-based LSTM decoder increases CIDEr-D from 120.1% to 127.2% on the COCO Karpathy test split.
• When combined with GCN to model semantic relations, the CIDEr-D score reaches 130.6%, setting a new state-of-the-art on COCO.
• Human evaluation shows that 76.5% of GCN-LSTM+HIP captions are as good or better than human-annotated captions, and 85.2% pass the Turing Test.
• The optimal threshold parameter ε for hierarchy construction is 0.1, as it maximizes performance on both METEOR and CIDEr-D.
• On the recognition task, HIP features achieve 2% and 3.2% gains in C-F1 and O-F1 over mean-pooled features from Up-Down, demonstrating generalizability.
• The ablation study confirms that the hierarchical Tree-LSTM and GCN-enhanced features are both critical for performance gains.