[Paper Review] Referring Image Segmentation Using Text Supervision

Fang Liu, Yuhao Liu+6 more|arXiv (Cornell University)|Aug 28, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes a weakly-supervised referring image segmentation framework that trains using only referring text descriptions, eliminating the need for expensive pixel- or box-level annotations. By formulating localization as a text classification task between positive (correct) and negative (distant) text expressions, the method uses a bilateral prompt, response map calibration, and positive response map selection to generate high-quality pseudo-labels, achieving performance competitive with fully-supervised methods on four benchmarks.

ABSTRACT

Existing Referring Image Segmentation (RIS) methods typically require expensive pixel-level or box-level annotations for supervision. In this paper, we observe that the referring texts used in RIS already provide sufficient information to localize the target object. Hence, we propose a novel weakly-supervised RIS framework to formulate the target localization problem as a classification process to differentiate between positive and negative text expressions. While the referring text expressions for an image are used as positive expressions, the referring text expressions from other images can be used as negative expressions for this image. Our framework has three main novelties. First, we propose a bilateral prompt method to facilitate the classification process, by harmonizing the domain discrepancy between visual and linguistic features. Second, we propose a calibration method to reduce noisy background information and improve the correctness of the response maps for target object localization. Third, we propose a positive response map selection strategy to generate high-quality pseudo-labels from the enhanced response maps, for training a segmentation network for RIS inference. For evaluation, we propose a new metric to measure localization accuracy. Experiments on four benchmarks show that our framework achieves promising performances to existing fully-supervised RIS methods while outperforming state-of-the-art weakly-supervised methods adapted from related areas. Code is available at https://github.com/fawnliu/TRIS.

Motivation & Objective

• To address the high annotation cost of fully-supervised referring image segmentation (RIS) by eliminating the need for pixel- or box-level annotations.
• To leverage existing referring text descriptions as the sole supervision signal for training RIS models.
• To improve localization accuracy in weakly-supervised RIS by reducing background noise and enhancing response map quality.
• To develop a new evaluation metric that reduces in-box errors compared to standard pointing-game metrics.
• To demonstrate that high-quality segmentation can be achieved using only text expressions as supervision, without additional annotations.

Proposed method

• Formulates RIS as a binary classification task: distinguishing referring texts that describe the target object (positive) from those describing objects in other images (negative).
• Introduces a bilateral prompt method to align visual and linguistic features by applying prompts to both modality streams, reducing domain discrepancy.
• Proposes a calibration method that enhances foreground response and suppresses background noise in the response maps using a learnable attention mechanism.
• Designs a positive response map selection (PRMS) strategy to select the most discriminative response map from multiple queries describing the same object, generating high-quality pseudo-labels.
• Trains a segmentation network using the selected pseudo-labels in a two-stage pipeline: first generating response maps, then refining segmentation predictions.
• Employs a new localization metric (PointM) that reduces in-box errors by measuring the distance between predicted and ground-truth response points.

Experimental results

Research questions

• RQ1 Can referring image segmentation be effectively trained using only text descriptions as supervision, without any additional annotations?
• RQ2 How can the model leverage inter-image text comparisons to improve localization robustness and reduce false positives?
• RQ3 To what extent can response map calibration and positive map selection improve pseudo-label quality and downstream segmentation performance?
• RQ4 How does the proposed PointM metric better evaluate localization accuracy compared to existing pointing-game metrics?
• RQ5 What is the optimal number of negative queries and calibration parameters for balancing model performance and generalization?

Key findings

• The proposed method achieves competitive performance on four RIS benchmarks—matching or exceeding state-of-the-art fully-supervised methods—while being trained with only text supervision.
• On the RefCOCO+ benchmark, the method achieves a 24.48% IoU, representing a 2.17% absolute improvement over the baseline without positive response map selection (22.31%).
• The positive response map selection (PRMS) strategy improves IoU by approximately 10% on the challenging RefCOCO+ set, demonstrating its effectiveness in selecting optimal response maps.
• The method performs best with 48 negative queries and K=6 in the calibration module, showing that performance degrades with excessive negative samples due to class imbalance.
• The new PointM metric reduces in-box errors compared to the standard pointing-game metric, providing a more accurate evaluation of localization precision.
• Failure cases reveal limitations in handling similar-semantic background objects or occluded targets, suggesting potential for future work with structured linguistic reasoning.