[Paper Review] Position-Aware Contrastive Alignment for Referring Image Segmentation

Bo Chen, Zhiwei Hu+3 more|arXiv (Cornell University)|Dec 27, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Position-Aware Contrastive Alignment Network (PCAN) for referring image segmentation, which enhances multi-modal feature alignment by integrating language-related object position information and contrastive learning. By leveraging GLIP-generated object locations and a dual-decoder contrastive module, PCAN improves segmentation accuracy without inference overhead, outperforming SOTA methods on three benchmarks.

ABSTRACT

Referring image segmentation aims to segment the target object described by a given natural language expression. Typically, referring expressions contain complex relationships between the target and its surrounding objects. The main challenge of this task is to understand the visual and linguistic content simultaneously and to find the referred object accurately among all instances in the image. Currently, the most effective way to solve the above problem is to obtain aligned multi-modal features by computing the correlation between visual and linguistic feature modalities under the supervision of the ground-truth mask. However, existing paradigms have difficulty in thoroughly understanding visual and linguistic content due to the inability to perceive information directly about surrounding objects that refer to the target. This prevents them from learning aligned multi-modal features, which leads to inaccurate segmentation. To address this issue, we present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features by guiding the interaction between vision and language through prior position information. Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment by comparing the features of the referred object with those of related objects. Extensive experiments on three benchmarks demonstrate our PCAN performs favorably against the state-of-the-art methods. Our code will be made publicly available.

Motivation & Objective

• To address the challenge of inaccurate segmentation in referring image segmentation due to poor understanding of visual-linguistic relationships and surrounding objects.
• To improve multi-modal feature alignment by incorporating prior knowledge of object positions related to linguistic descriptions.
• To reduce inference cost while maintaining high performance by transferring contrastive learning knowledge to the matching head.
• To enhance model generalization and robustness in complex scenes with ambiguous or similar objects.
• To develop a method that leverages pre-trained vision-language detectors for better localization guidance without additional detection heads.

Proposed method

• The Position-Aware Module (PAM) uses GLIP to extract high-quality object regions related to the referring expression, providing prior position information.
• The Contrastive Language Understanding Module (CLUM) employs two weight-shared decoders: one for contrastive learning using language-related object positions, and one for matching using random queries.
• In CLUM, contrastive learning transfers robust feature representations from the contrastive decoder to the matching decoder, enhancing query feature quality.
• The model uses a dual-branch cross-attention mechanism to align visual and linguistic features, guided by position-aware queries.
• The contrastive loss is applied between the referred object and related but non-target objects to improve discrimination and alignment.
• During inference, only the matching decoder is used, eliminating the need for external detectors and avoiding additional computational cost.

Experimental results

Research questions

• RQ1 Can incorporating language-related object position information improve multi-modal feature alignment in referring image segmentation?
• RQ2 How does contrastive learning based on position-aware features enhance the model's ability to distinguish the referred object from similar distractors?
• RQ3 Does integrating prior object position information lead to better segmentation performance without increasing inference complexity?
• RQ4 What is the optimal strategy for generating positive and negative samples in the contrastive learning module?
• RQ5 How do the number of object proposals and grouping strategies affect the model's performance and generalization?

Key findings

• PCAN achieves state-of-the-art performance on three standard benchmarks: RefCOCO, RefCOCO+, and Ref-YOLO, with significant improvements over prior methods.
• Using GLIP-generated boxes as negative samples yields a 0.61% performance gain over random negative sampling, demonstrating the effectiveness of relevant object priors.
• The conditional negative sampling strategy improves performance by 0.47% over unconditional sampling, indicating better model guidance.
• The model achieves optimal performance with K=6 object proposals and G=3 groups, with performance saturating beyond these settings.
• Abllation studies confirm that combining PAM and CLUM yields the best results, showing complementary benefits between position awareness and contrastive learning.
• Visualization results show that the full model produces more accurate and robust segmentation masks, especially in complex scenes with similar objects.