[Paper Review] The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation

Sergi Caelles, Jordi Pont-Tuset+4 more|arXiv (Cornell University)|May 2, 2019

Visual Attention and Saliency DetectionComputer Science45 references100 citations

TL;DR

Introduces an unsupervised multi-object video object segmentation track for DAVIS, re-annotates DAVIS 2017 train/val for consistency, and provides baseline RVOS results on new test sets.

ABSTRACT

We present the 2019 DAVIS Challenge on Video Object Segmentation, the third edition of the DAVIS Challenge series, a public competition designed for the task of Video Object Segmentation (VOS). In addition to the original semi-supervised track and the interactive track introduced in the previous edition, a new unsupervised multi-object track will be featured this year. In the newly introduced track, participants are asked to provide non-overlapping object proposals on each image, along with an identifier linking them between frames (i.e. video object proposals), without any test-time human supervision (no scribbles or masks provided on the test video). In order to do so, we have re-annotated the train and val sets of DAVIS 2017 in a concise way that facilitates the unsupervised track, and created new test-dev and test-challenge sets for the competition. Definitions, rules, and evaluation metrics for the unsupervised track are described in detail in this paper.

Motivation & Objective

• Motivate the introduction of an unsupervised multi-object VOS track to cover fully automatic scenarios.
• Provide a precise definition of unsupervised multi-object VOS and the criteria for object selection and grouping.
• Offer evaluation metrics and rules to consistently assess unsupervised proposals across sequences.
• Re-annotate DAVIS 2017 train/val to align with unsupervised guidelines and publish new test-dev/challenge sets.
• Establish baselines (e.g., RVOS) to gauge the difficulty of unsupervised multi-object VOS.

Proposed method

• Define unsupervised multi-object VOS with semantics-focused object selection and consistent mask tracking across frames.
• Propose a bipartite matching formulation using a J&F-based accuracy matrix and solve with the Hungarian algorithm to assign predicted video object proposals to ground-truth objects.
• Use a pool of non-overlapping video object proposals (N per sequence) and match each ground-truth object to a single proposal maximizing M(O_gt, O_n).
• Adopt the J&F metric for evaluation, mirroring semi-supervised practices, to compute per-object and per-sequence scores.
• Utilize RVOS as a baseline in zero-shot mode (20 proposals per sequence) to quantify unsupervised performance.
• Re-annotate DAVIS 2017 train/val to ensure consistency with the unsupervised definition and introduce new test-dev/test-challenge sets for the unsupervised track.

Experimental results

Research questions

• RQ1 How should objects be selected and grouped for unsupervised multi-object VOS to reflect human attention and semantic boundaries?
• RQ2 What evaluation framework (metrics and matching) best measures unsupervised multi-object VOS performance?
• RQ3 How does a zero-shot video object proposal method (e.g., RVOS) perform on the re-annotated DAVIS 2017 Unsupervised data?
• RQ4 What is the gap between unsupervised and semi-supervised VOS performance on DAVIS-derived benchmarks?

Key findings

• A new unsupervised multi-object track is defined and evaluated within the DAVIS framework.
• The authors re-annotated DAVIS 2017 train/val to align with unsupervised semantics and added test-dev/test-challenge for the unsupervised track.
• RVOS in zero-shot mode provides baseline results, illustrating the increased challenge of unsupervised multi-object VOS compared to semi-supervised settings.
• On DAVIS 2017 Unsupervised, RVOS achieves 41.2 J&F and 36.8 J Mean on val, and 22.5 J&F and 17.7 J Mean on test-dev; F1 and related recalls/decays are reported as well.
• The evaluation uses a maximum bipartite matching (Hungarian algorithm) to assign predicted proposals to ground-truth objects, maximizing an accuracy matrix M across L ground-truth objects and N proposals.
• Multi-object unsupervised VOS remains substantially more challenging than semi-supervised VOS, as indicated by the reported baseline performance.