[Paper Review] YouTube-VOS: Sequence-to-Sequence Video Object Segmentation

Ning Xu, Linjie Yang+7 more|arXiv (Cornell University)|Sep 3, 2018

Advanced Image and Video Retrieval Techniques36 references20 citations

TL;DR

This paper introduces YouTube-VOS, the largest video object segmentation dataset to date with 3,252 YouTube clips and 78 categories, enabling end-to-end sequence-to-sequence learning for long-term spatial-temporal modeling. The proposed method uses a convolutional LSTM-based sequence-to-sequence network that directly learns temporal dependencies without relying on pre-trained optical flow or motion models, achieving state-of-the-art performance on YouTube-VOS and competitive results on DAVIS 2016.

ABSTRACT

Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatial-temporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 3,252 YouTube video clips and 78 categories including common objects and human activities. This is by far the largest video object segmentation dataset to our knowledge and we have released it at https://youtube-vos.org. Based on this dataset, we propose a novel sequence-to-sequence network to fully exploit long-term spatial-temporal information in videos for segmentation. We demonstrate that our method is able to achieve the best results on our YouTube-VOS test set and comparable results on DAVIS 2016 compared to the current state-of-the-art methods. Experiments show that the large scale dataset is indeed a key factor to the success of our model.

Motivation & Objective

• To address the lack of large-scale video segmentation datasets that hinder end-to-end learning of long-term spatial-temporal features.
• To overcome the limitations of existing video object segmentation methods that rely on pre-trained optical flow or motion models, which are suboptimal for segmentation.
• To develop a novel sequence-to-sequence deep learning framework that fully exploits long-term temporal dependencies in video for accurate object segmentation.
• To evaluate the impact of dataset scale on model performance and demonstrate the necessity of large-scale data for training robust video segmentation models.

Proposed method

• Proposes a sequence-to-sequence network using a convolutional LSTM (ConvLSTM) to model long-term spatial-temporal features across video frames.
• At each time step, the ConvLSTM takes the encoded image frame and the previous hidden state to produce updated spatiotemporal features for mask decoding.
• Employs a VGG-16-based Initializer to generate initial hidden states from the first frame's RGB image and object mask.
• Uses teacher forcing during training, where ground-truth masks from the previous frame are used as input to prevent error accumulation, transitioning to self-supervised inference.
• Introduces a variant that replaces the Initializer with direct mask input to evaluate its effectiveness, showing inferior performance.
• Explores an encoder variant that uses both the RGB frame and the previous frame’s prediction mask as input, improving training stability and performance.

Experimental results

Research questions

• RQ1 Can a large-scale video segmentation dataset significantly improve the performance of end-to-end sequence-to-sequence models?
• RQ2 How effective is a ConvLSTM-based sequence-to-sequence network in modeling long-term spatial-temporal dependencies without relying on pre-trained optical flow models?
• RQ3 What is the impact of training data scale on the generalization and performance of video object segmentation models?
• RQ4 Can direct mask initialization or mask-informed encoding improve segmentation accuracy compared to learned initial hidden states?
• RQ5 How does the combination of teacher forcing and curriculum learning strategies affect training stability and final performance?

Key findings

• The proposed sequence-to-sequence model achieves a mean Jaccard index (J) of 60.9% and F-measure of 64.2% on the YouTube-VOS test set, outperforming existing state-of-the-art methods.
• On the DAVIS 2016 benchmark, the model achieves comparable performance to state-of-the-art methods, demonstrating strong generalization.
• Training with only 25% of the YouTube-VOS training data results in a 30% drop in performance, highlighting the critical role of dataset scale.
• Models trained on 100% of the YouTube-VOS data show no performance plateau, indicating that larger data could further improve results.
• The model generalizes well to unseen categories, achieving 60.7% Jaccard index on unseen categories, suggesting effective learning of general object features.
• Replacing the Initializer with direct mask input reduces performance to 45.1% J, indicating that the mask alone lacks sufficient representational capacity for initialization.