[Paper Review] 1st Place Solution for YouTubeVOS Challenge 2021:Video Instance Segmentation

Nguyễn Thúy Chinh, Tuan N. Tang+4 more|arXiv (Cornell University)|Jun 11, 2021

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper presents a first-place solution for the YouTube-VIS 2021 challenge, introducing a unified video instance segmentation model that leverages Temporally Correlated Instance Segmentation (TCIS), Multi-Source Data (MSD) training to combat frame redundancy, and Bidirectional Tracking (BiTrack) for improved object recall. The method achieves state-of-the-art performance with 0.575 mAP on the public validation set and 0.541 mAP on the private test set, setting new records on both YouTube-VIS 2019 and 2021 benchmarks.

ABSTRACT

Video Instance Segmentation (VIS) is a multi-task problem performing detection, segmentation, and tracking simultaneously. Extended from image set applications, video data additionally induces the temporal information, which, if handled appropriately, is very useful to identify and predict object motions. In this work, we design a unified model to mutually learn these tasks. Specifically, we propose two modules, named Temporally Correlated Instance Segmentation (TCIS) and Bidirectional Tracking (BiTrack), to take the benefit of the temporal correlation between the object's instance masks across adjacent frames. On the other hand, video data is often redundant due to the frame's overlap. Our analysis shows that this problem is particularly severe for the YoutubeVOS-VIS2021 data. Therefore, we propose a Multi-Source Data (MSD) training mechanism to compensate for the data deficiency. By combining these techniques with a bag of tricks, the network performance is significantly boosted compared to the baseline, and outperforms other methods by a considerable margin on the YoutubeVOS-VIS 2019 and 2021 datasets.

Motivation & Objective

• To address the challenge of high frame redundancy and low data efficiency in video instance segmentation datasets like YouTube-VIS 2021.
• To improve performance in video instance segmentation by jointly learning detection, segmentation, and tracking through a unified model.
• To enhance model generalization and robustness using a novel Multi-Source Data (MSD) training mechanism that increases data diversity.
• To improve tracking accuracy and recall by introducing Bidirectional Tracking (BiTrack), which processes video forward and backward to recover missed objects.
• To achieve state-of-the-art performance on YouTube-VIS 2019 and 2021 benchmarks through a combination of architectural innovations and practical training tricks.

Proposed method

• Proposes Temporally Correlated Instance Segmentation (TCIS), a module that explicitly models temporal dependencies between instance masks in adjacent frames to improve consistency and accuracy.
• Introduces Multi-Source Data (MSD), a training mechanism that combines data from multiple sources—including different video segments and frame sampling strategies—to increase data diversity and reduce overfitting to redundant frames.
• Employs Bidirectional Tracking (BiTrack), a post-processing step that runs tracking forward and backward through the video, then merges the results to recover more complete object trajectories.
• Uses a Mask R-CNN-based backbone with a 6-level Feature Pyramid Network (FPN) and additional P7 feature level to improve multi-scale feature representation.
• Applies a multi-task learning framework with shared features for detection, segmentation, and tracking, including auxiliary heads for mask scoring and semantic segmentation.
• Employs ensemble learning and pseudo-labeling on the validation set to further boost performance, especially on the private test set.

Experimental results

Research questions

• RQ1 How can temporal correlations between adjacent frames be effectively exploited to improve video instance segmentation performance?
• RQ2 To what extent does frame redundancy in YouTube-VIS 2021 degrade model performance, and how can data efficiency be improved?
• RQ3 Can bidirectional tracking improve object recall and trajectory completeness in video instance segmentation?
• RQ4 How do architectural components like TCIS and MSD contribute to generalization and robustness in low-data or redundant data regimes?
• RQ5 What combination of training techniques and model ensembling yields the highest performance on the YouTube-VIS 2021 and 2019 benchmarks?

Key findings

• The proposed method achieves 0.575 mAP on the YouTube-VIS 2021 public validation set, securing the first-place rank and setting a new state-of-the-art benchmark.
• On the private test set, the method achieves 0.541 mAP, demonstrating strong generalization and robustness beyond the validation set.
• The model achieves 0.543 mAP on the YouTube-VIS 2019 benchmark, significantly outperforming the previous state-of-the-art method (MaskProp) by 11.0% in mAP.
• The ablation study shows that TCIS, MSD, and BiTrack each contribute over 2% mAP improvement individually, with combined components boosting performance by approximately 8% mAP over the baseline.
• The bag of tricks—including larger backbones (ResNeSt101, Swin-S), pseudo-labeling, and model ensembling—leads to a 15%+ mAP gain over the standard backbone, highlighting the importance of practical training strategies.
• Visual results confirm that the model accurately detects, segments, and tracks objects across frames with consistent instance masks and color-coded identities.