[Paper Review] Referring Expression Object Segmentation with Caption-Aware Consistency

Yiwen Chen, Yi‐Hsuan Tsai+3 more|arXiv (Cornell University)|Oct 10, 2019

Multimodal Machine Learning Applications27 references8 citations

TL;DR

This paper proposes a novel end-to-end framework for referring expression object segmentation that jointly optimizes comprehension and generation via caption-aware consistency. By introducing spatial-aware dynamic filters to transfer language knowledge to visual features and enforcing consistency between input and generated referring expressions, the method achieves state-of-the-art performance on RefCOCO and RefCOCO+ datasets.

ABSTRACT

Referring expressions are natural language descriptions that identify a particular object within a scene and are widely used in our daily conversations. In this work, we focus on segmenting the object in an image specified by a referring expression. To this end, we propose an end-to-end trainable comprehension network that consists of the language and visual encoders to extract feature representations from both domains. We introduce the spatial-aware dynamic filters to transfer knowledge from text to image, and effectively capture the spatial information of the specified object. To better communicate between the language and visual modules, we employ a caption generation network that takes features shared across both domains as input, and improves both representations via a consistency that enforces the generated sentence to be similar to the given referring expression. We evaluate the proposed framework on two referring expression datasets and show that our method performs favorably against the state-of-the-art algorithms.

Motivation & Objective

• To improve referring expression object segmentation by jointly modeling comprehension and generation tasks.
• To enhance cross-modal feature learning between language and vision using a consistency loss between input and generated referring expressions.
• To design spatial-aware dynamic filters that transfer linguistic knowledge to visual features with spatial localization guidance.
• To achieve end-to-end training with improved localization accuracy and robustness to complex scenes.

Proposed method

• A comprehension network with separate language and visual encoders to extract modality-specific features.
• Spatial-aware dynamic filters that generate convolutional filters from text features to attend to relevant image regions.
• A caption generation network that takes shared features as input and produces a referring expression to enforce consistency with the input.
• A caption-aware consistency loss that minimizes the difference between the input referring expression and the generated sentence.
• End-to-end training using cross-entropy and consistency losses to jointly optimize segmentation and generation.
• Integration of spatial coordinates into feature maps to improve localization precision in dynamic filtering.

Experimental results

Research questions

• RQ1 Can joint optimization of referring expression comprehension and generation improve segmentation performance?
• RQ2 How does enforcing consistency between input and generated referring expressions affect feature representation quality?
• RQ3 Can spatial-aware dynamic filters better localize target objects than standard or non-spatial dynamic filters?
• RQ4 What is the contribution of each component—spatial-aware filters and caption consistency—to overall performance?

Key findings

• The proposed method achieves 58.90% and 61.77% mask intersection-over-union (mIoU) on RefCOCO, outperforming state-of-the-art methods.
• On RefCOCO+, the model obtains 53.81% mIoU, demonstrating strong generalization across datasets.
• Ablation studies confirm that both spatial-aware dynamic filters and caption-aware consistency significantly improve performance over baseline models.
• The full model with both components achieves 46.95% mIoU on RefCOCOg, surpassing existing methods including MAttNet and DMN.
• Qualitative results show accurate segmentation of objects based on complex descriptions involving color, location, and action.
• The model effectively segments objects even in cluttered scenes, such as the 'elephant in back' example, where background confusion is common.