[Paper Review] The 2018 DAVIS Challenge on Video Object Segmentation

Sergi Caelles, Alberto Montes+5 more|arXiv (Cornell University)|Mar 1, 2018

Visual Attention and Saliency DetectionComputer Science13 references254 citations

TL;DR

The paper introduces the 2018 DAVIS Challenge with a semi-supervised main track and a new interactive segmentation teaser track, along with interactive baselines and a simulated scribble framework for evaluation.

ABSTRACT

We present the 2018 DAVIS Challenge on Video Object Segmentation, a public competition specifically designed for the task of video object segmentation. It builds upon the DAVIS 2017 dataset, which was presented in the previous edition of the DAVIS Challenge, and added 100 videos with multiple objects per sequence to the original DAVIS 2016 dataset. Motivated by the analysis of the results of the 2017 edition, the main track of the competition will be the same than in the previous edition (segmentation given the full mask of the objects in the first frame -- semi-supervised scenario). This edition, however, also adds an interactive segmentation teaser track, where the participants will interact with a web service simulating the input of a human that provides scribbles to iteratively improve the result.

Motivation & Objective

• Motivate and organize a public benchmark for video object segmentation based on the DAVIS dataset extension from 2017.
• Provide a semi-supervised main track using first-frame masks to segment all subsequent frames.
• Introduce an interactive segmentation teaser track with a simulated human-in-the-loop to study efficiency–quality trade-offs.
• Propose evaluation metrics and baselines to encourage realistic, user-in-the-loop segmentation research.

Proposed method

• Extend the DAVIS 2017 dataset by adding 100 videos with multiple objects per sequence for richer evaluation.
• Retain the semi-supervised main track (mask in the first frame) while adding an interactive segmentation pipeline evaluated via a web service.
• Simulate human scribbles through an automatic robot to enable scalable interactive evaluation while keeping test annotations private.
• Propose two interactive baselines: Scribble-OSVOS (fine-tune on scribble-derived labels) and Scribble-Linear (CNN embeddings with a linear classifier).
• Evaluate trade-offs between interaction time and segmentation quality using J&F as the metric, across quality and speed tracks.

Experimental results

Research questions

• RQ1 How does interactive, scribble-based supervision compare to traditional semi-supervised approaches in terms of quality versus time?
• RQ2 Can simulated scribbles reliably predict human-in-the-loop performance for interactive video object segmentation baselines?
• RQ3 Do baseline methods approached with scribble supervision reach fully supervised performance with reduced labeling effort?

Key findings

• Scribble-OSVOS with scribble supervision reaches 57% J&F in 1200 seconds under full supervision, surpassing the full-supervision baseline in faster time.
• Scribble-OSVOS (using scribbles) achieves 39.26% J&F in 110 seconds for the DAVIS 2017 validation set, bypassing long retraining times.
• A simulated scribble generator yields conclusions similar to manually drawn scribbles for both baselines, supporting its use for scalable interactive evaluation.
• A faster Scribble-Linear baseline achieves comparable trends to Scribble-OSVOS, highlighting the viability of embedding-based interactive approaches.
• The experiments show interactive methods can reach or approach fully supervised performance with substantially less labeling time.