[Paper Review] YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark

Ning Xu, Linjie Yang+5 more|arXiv (Cornell University)|Sep 6, 2018

Visual Attention and Saliency DetectionComputer Science26 references302 citations

TL;DR

Introduces a large-scale YouTube-VOS dataset for video object segmentation and benchmarks multiple state-of-the-art methods on its validation set to establish baselines and analyze generalization.

ABSTRACT

Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatialtemporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 4,453 YouTube video clips and 94 object categories. This is by far the largest video object segmentation dataset to our knowledge and has been released at http://youtube-vos.org. We further evaluate several existing state-of-the-art video object segmentation algorithms on this dataset which aims to establish baselines for the development of new algorithms in the future.

Motivation & Objective

• Motivate end-to-end learning of long-term spatial-temporal features for video object segmentation due to limitations of existing small-scale datasets.
• Create a large-scale, diverse dataset from YouTube videos to enable robust training and evaluation of sequential models.
• Provide a benchmark to evaluate generalization to unseen categories and establish baseline performances for future work.

Proposed method

• Assemble a new large-scale video object segmentation dataset from YouTube videos across 94 categories with dense annotations every five frames (6 fps sampling).
• Leverage a skip-frame annotation strategy to scale annotations while preserving temporal coherence across frames.
• Evaluate several state-of-the-art video object segmentation methods on YouTube-VOS training/validation split under consistent settings.
• Analyze performance on seen vs unseen categories to assess generalization.
• Provide baseline results and insights into the effectiveness of online learning and long-term temporal modeling for segmentation.

Experimental results

Research questions

• RQ1 How does a large-scale YouTube-derived dataset impact the training of video object segmentation models compared to smaller benchmarks?
• RQ2 What is the performance gap between seen and unseen categories, and how does online learning affect generalization?
• RQ3 How do long-term spatial-temporal models compare to static-image-based approaches on YouTube-VOS?
• RQ4 What are the trade-offs between inference speed and accuracy across methods on the YouTube-VOS validation set?

Key findings

• YouTube-VOS is the largest VOS dataset to date with 4,453 videos and 197,272 annotations across 94 object categories.
• Sequence-to-sequence models with temporal coherence (S2S) outperform static-image methods, especially when using online learning (OL).
• OnAVOS underperforms compared to expectations from DAVIS benchmarks due to drastic appearance changes and complex motion in YouTube-VOS.
• In unseen categories, all methods degrade, but OSVOS shows relatively smaller decay, indicating benefit from large-scale image pre-training.
• Inference-speed-optimized methods (OSMN, S2S w/o OL) offer real-time potential, though with lower accuracy than OL variants.