[Paper Review] Learning Video Object Segmentation from Unlabeled Videos

Xiankai Lu, Wenguan Wang+4 more|arXiv (Cornell University)|Mar 10, 2020

Visual Attention and Saliency DetectionComputer Science70 references20 citations

TL;DR

This paper proposes MuG, a unified unsupervised/weakly supervised video object segmentation framework that learns video object patterns from unlabeled videos by leveraging multi-granularity cues—frame-level, short-term, long-range, and whole-video—without requiring pixel-level annotations. The method achieves state-of-the-art performance across object-level zero-shot VOS, instance-level zero-shot VOS, and one-shot VOS settings, significantly reducing annotation burden while maintaining high accuracy.

ABSTRACT

We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos, unlike most existing methods which rely heavily on extensive annotated data. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures intrinsic properties of VOS at multiple granularities. Our approach can help advance understanding of visual patterns in VOS and significantly reduce annotation burden. With a carefully-designed architecture and strong representation learning ability, our learned model can be applied to diverse VOS settings, including object-level zero-shot VOS, instance-level zero-shot VOS, and one-shot VOS. Experiments demonstrate promising performance in these settings, as well as the potential of MuG in leveraging unlabeled data to further improve the segmentation accuracy.

Motivation & Objective

• To reduce the heavy reliance on expensive pixel-wise annotations in video object segmentation (VOS) by enabling learning from unlabeled videos.
• To unify diverse VOS settings—object-level zero-shot, instance-level zero-shot, and one-shot VOS—under a single unsupervised/weakly supervised framework.
• To model intrinsic video object patterns by exploiting multi-granularity supervision signals from unlabeled data, including saliency maps, CAM activations, and temporal consistency.
• To advance understanding of visual patterns in VOS by mining heuristics and structural properties inherent in video data.

Proposed method

• MuG employs a unified architecture that integrates multi-granularity supervision signals: frame-level cues from unsupervised saliency or CAM maps, short-term consistency in clip-level representations, long-range correspondence across distant frames, and global video-level compactness.
• The model uses a self-supervised contrastive learning objective to align representations across frames, enhancing robustness to occlusions and appearance variations.
• It formulates a unified optimization objective that combines multiple constraints at different granularities, enabling end-to-end training without ground-truth masks.
• The method leverages weak supervision from pre-trained image classifiers via class activation maps (CAMs) and saliency predictions as auxiliary supervision signals.
• A multi-stage training strategy progressively refines representations by first learning local motion and appearance patterns, then enforcing long-range consistency.
• The framework is designed to be adaptable to various VOS settings, including zero-shot and one-shot inference, by leveraging the learned video object patterns.

Experimental results

Research questions

• RQ1 Can video object segmentation be effectively learned from unlabeled videos without relying on expensive pixel-level annotations?
• RQ2 How can multi-granularity cues—frame-level, short-term, long-range, and whole-video—be jointly exploited to model intrinsic video object patterns?
• RQ3 Can a unified unsupervised/weakly supervised framework generalize across diverse VOS settings, including object-level and instance-level zero-shot VOS and one-shot VOS?
• RQ4 To what extent can weak supervision from CAMs or saliency maps improve segmentation performance in the absence of ground-truth masks?
• RQ5 How does the proposed method compare to supervised and self-supervised baselines in terms of accuracy and generalization across different datasets and settings?

Key findings

• MuG achieves state-of-the-art performance on object-level zero-shot VOS, outperforming previous unsupervised methods by +14.8% in mean J&F on DAVIS 17.
• In instance-level zero-shot VOS, MuG achieves competitive performance comparable to fully supervised methods like AGS and PDB, significantly outperforming RVOS by +14.8% and +19.2% in mean J&F under unsupervised and weakly-supervised settings.
• For one-shot VOS on DAVIS 17, MuG achieves a mean J&F of 54.3, surpassing the previous SOTA method CorrFlow (50.3), demonstrating superior pattern learning from unlabeled data.
• The model processes each frame in approximately 0.7s for object-level Z-VOS and 0.4s for instance-level Z-VOS, making it efficient compared to prior self-supervised and matching-based methods.
• Visual results show robust performance under challenging conditions such as view changes, background clutter, fast motion, and scale variation, confirming the model’s generalization capability.
• The ablation study confirms that all four granularity levels—frame, short-term, long-range, and whole-video—contribute significantly to the final performance, validating the multi-granularity design.