[Paper Review] Cross-Modal Self-Attention Network for Referring Image Segmentation

Linwei Ye, Mrigank Rochan+2 more|arXiv (Cornell University)|Apr 9, 2019

Multimodal Machine Learning Applications31 references46 citations

TL;DR

The paper introduces a cross-modal self-attention (CMSA) module to capture long-range dependencies between language and vision for referring image segmentation, plus a gated multi-level fusion to integrate multi-scale features, achieving state-of-the-art results on four datasets.

ABSTRACT

We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the input image. In addition, we propose a gated multi-level fusion module to selectively integrate self-attentive cross-modal features corresponding to different levels in the image. This module controls the information flow of features at different levels. We validate the proposed approach on four evaluation datasets. Our proposed approach consistently outperforms existing state-of-the-art methods.

Motivation & Objective

• Motivate accurate segmentation of objects described by natural language in images, beyond simple category-based cues.
• Capture long-range dependencies between linguistic and visual features to improve referring segmentation.
• Develop a mechanism to integrate multi-level CNN features in a selective, information-flow controlled manner.

Proposed method

• Construct multimodal features by combining image features, word embeddings, and 8-D spatial coordinates for each image location and word.
• Apply a cross-modal self-attention (CMSA) module to learn long-range dependencies across words and spatial regions, using learned queries, keys, and values with residual connections.
• Aggregate CMSA outputs by average-pooling across words to obtain per-location multimodal features.
• Introduce a gated multi-level fusion (GF) module that uses 1x1 convolutions, memory and reset gates, and contextual controllers to selectively fuse features from three CNN levels (Res3, Res4, Res5).
• Produce the final segmentation mask via 3x3 convolution and sigmoid, trained with a binary cross-entropy loss and optimized with Adam.

Experimental results

Research questions

• RQ1 Can cross-modal self-attention effectively model long-range dependencies between language and vision for referring image segmentation?
• RQ2 Does a gated multi-level fusion mechanism improve the refinement and accuracy of segmentation masks across multiple CNN feature levels?
• RQ3 How does word-level attention compare to sentence-level encoding for multimodal segmentation performance?
• RQ4 What is the impact of multi-level feature fusion approaches on segmentation quality across standard benchmarks?

Key findings

• The CMSA approach consistently outperforms state-of-the-art methods on four benchmark datasets (UNC, UNC+, G-Ref, ReferIt).
• Word-level multimodal representations with CMSA outperform sentence-based encodings and other attention variants in ablations.
• The gated multi-level fusion module improves integration of multi-scale CMSA features, outperforming Deconv, PPM, ConvLSTM, and simple gating baselines.
• Qualitative analyses show level-specific word attention focusing on attributes, relationships, or object nouns, and spatial heatmaps responding to diverse queries.
• Ablation studies confirm the superiority of cross-modal self-attention and gated fusion over individual components.