[Paper Review] Learning Referring Video Object Segmentation from Weak Annotation

Wangbo Zhao, Kepan Nan+4 more|arXiv (Cornell University)|Aug 4, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes a weakly supervised annotation scheme for referring video object segmentation (RVOS) that reduces labeling cost by 8× by using only a single mask for the first appearance of the target object and bounding boxes for subsequent frames. The method, SimRVOS with language-guided cross-frame segmentation and bi-level contrastive learning, achieves performance comparable to fully supervised methods, demonstrating that high-quality segmentation can be learned with minimal dense annotations.

ABSTRACT

Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. Although existing RVOS methods have achieved significant performance, they depend on densely-annotated datasets, which are expensive and time-consuming to obtain. In this paper, we propose a new annotation scheme that reduces the annotation effort by 8 times, while providing sufficient supervision for RVOS. Our scheme only requires a mask for the frame where the object first appears and bounding boxes for the rest of the frames. Based on this scheme, we develop a novel RVOS method that exploits weak annotations effectively. Specifically, we build a simple but effective baseline model, SimRVOS, for RVOS with weak annotation. Then, we design a cross frame segmentation module, which uses the language-guided dynamic filters from one frame to segment the target object in other frames to thoroughly leverage the valuable mask annotation and bounding boxes. Finally, we develop a bi-level contrastive learning method to enhance the pixel-level discriminative representation of the model with weak annotation. We conduct extensive experiments to show that our method achieves comparable or even superior performance to fully-supervised methods, without requiring dense mask annotations.

Motivation & Objective

• To reduce the high annotation cost of dense mask annotations in referring video object segmentation (RVOS) while maintaining high performance.
• To design a practical and efficient annotation scheme that minimizes human labeling effort without sacrificing segmentation accuracy.
• To develop a deep learning framework that effectively leverages weakly supervised signals—specifically, one mask and bounding boxes across frames—for end-to-end RVOS.
• To improve pixel-level feature discrimination under weak supervision using a novel bi-level contrastive learning strategy.

Proposed method

• Proposes a new annotation scheme: a single mask for the first frame where the target appears, and bounding boxes for all other frames, reducing annotation cost by 8×.
• Introduces SimRVOS, a simple yet effective baseline model that uses cross-modal fusion and language-guided dynamic convolution to generate segmentation masks.
• Designs a language-guided cross-frame segmentation (LGCFS) module that transfers dynamic filters from one frame to predict masks in others, leveraging both mask and bounding box supervision.
• Develops a bi-level contrastive learning (BLCL) method with two components: (a) language-vision contrast to align linguistic and foreground visual features, and (b) consistency contrast to enhance robustness across frames.
• Uses the LGCFS module to propagate mask supervision across frames while using bounding boxes as auxiliary supervision, improving generalization under appearance variation.
• Trains the model end-to-end using weak annotations, enabling strong performance without requiring dense pixel-level annotations.

Experimental results

Research questions

• RQ1 Can a referring video object segmentation model achieve competitive performance using only a single mask and bounding boxes across frames, rather than dense mask annotations?
• RQ2 How can valuable mask supervision from the first frame be effectively transferred and utilized across all video frames under weak supervision?
• RQ3 To what extent can bi-level contrastive learning improve pixel-level feature discrimination when only weak annotations are available?
• RQ4 Can a weakly supervised RVOS framework outperform or match the performance of fully supervised methods without dense annotations?

Key findings

• The proposed weak annotation scheme reduces labeling cost by 8× compared to fully dense annotation, as validated on YouTube-RVOS.
• The model achieves a mean IoU of 62.7 on the JHMDB-Sentences benchmark, outperforming fully supervised methods like LBDT and MTTR under weak supervision.
• SimRVOS with the proposed LGCFS and BLCL modules achieves 65.8% mAP on the JHMDB-Sentences set, surpassing the fully supervised LBDT method (65.8% mAP) and matching ReferFormer (69.3% mAP) in overall performance.
• The bi-level contrastive learning method significantly improves feature discriminativeness, as shown by consistent improvements in IoU and P@0.5 to P@0.9 metrics across multiple benchmarks.
• The LGCFS module enables effective mask propagation across frames, resulting in more accurate predictions than using bounding boxes alone, as shown in qualitative comparisons.
• The method generalizes well across datasets, achieving strong zero-shot performance on JHMDB-Sentences without fine-tuning, indicating robustness to domain shift.