[Paper Review] Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge

Jiyang Qi, Yan Gao+8 more|arXiv (Cornell University)|Nov 15, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces OVIS, a large-scale video instance segmentation dataset with 296k high-quality masks across 901 occluded video scenes, specifically designed to benchmark and advance occlusion reasoning in video understanding. Despite state-of-the-art methods, performance drops drastically—AP falls to 5.6 on heavily occluded objects—highlighting a critical gap in current models, which the authors address via an ICCV 2021 challenge that spurred new methods achieving up to 21.6 AP.

ABSTRACT

Although deep learning methods have achieved advanced video object recognition performance in recent years, perceiving heavily occluded objects in a video is still a very challenging task. To promote the development of occlusion understanding, we collect a large-scale dataset called OVIS for video instance segmentation in the occluded scenario. OVIS consists of 296k high-quality instance masks and 901 occluded scenes. While our human vision systems can perceive those occluded objects by contextual reasoning and association, our experiments suggest that current video understanding systems cannot. On the OVIS dataset, all baseline methods encounter a significant performance degradation of about 80% in the heavily occluded object group, which demonstrates that there is still a long way to go in understanding obscured objects and videos in a complex real-world scenario. To facilitate the research on new paradigms for video understanding systems, we launched a challenge based on the OVIS dataset. The submitted top-performing algorithms have achieved much higher performance than our baselines. In this paper, we will introduce the OVIS dataset and further dissect it by analyzing the results of baselines and submitted methods. The OVIS dataset and challenge information can be found at http://songbai.site/ovis .

Motivation & Objective

• To address the critical challenge of perceiving heavily occluded objects in videos, where current video understanding systems significantly underperform compared to human vision.
• To create a large-scale, high-quality dataset focused on occluded video instance segmentation to better reflect real-world complexity and diversity.
• To evaluate the limitations of existing state-of-the-art video instance segmentation models on occlusion-heavy scenarios through comprehensive baseline analysis.
• To stimulate research on new paradigms for occlusion reasoning by launching the ICCV 2021 Occluded Video Instance Segmentation Challenge.

Proposed method

• The OVIS dataset was constructed by collecting 901 complex video scenes with diverse occlusion types, including long sequences and high object density to maximize occlusion exposure.
• Each instance in every frame is annotated with precise instance masks and an occlusion level score, enabling fine-grained evaluation across occlusion degrees.
• Nine state-of-the-art video instance segmentation models were evaluated on OVIS to establish baseline performance, including MaskTrack R-CNN, STEm-Seg, and QueryInst.
• The ICCV 2021 challenge invited submissions using novel architectures and training strategies, with top methods leveraging temporal modeling and data augmentation from image-level datasets like COCO and Pascal VOC.
• Ablation studies were conducted to analyze the impact of data augmentation, such as synthesizing image pairs from COCO, on model robustness to occlusion.
• Performance was evaluated using standard metrics including AP (average precision), with separate scores for heavily occluded objects (AP_HO).

Experimental results

Research questions

• RQ1 How do current state-of-the-art video instance segmentation models perform on heavily occluded objects compared to human perception?
• RQ2 To what extent does occlusion degrade the performance of existing video instance segmentation methods on real-world complex scenes?
• RQ3 What architectural or training innovations enable improved performance on occluded video instance segmentation?
• RQ4 Can data augmentation using image-level instance segmentation datasets improve robustness to occlusion in video models?
• RQ5 What are the key failure modes of current models in occluded scenarios, and how do they compare to human reasoning in context and association?

Key findings

• All baseline methods suffered a drastic performance drop on heavily occluded objects, with the highest AP on this group reaching only 5.6, indicating a major gap in occlusion reasoning.
• The top-performing method in the challenge achieved an AP of 21.6 on the test set, significantly outperforming the best baseline (16.3 AP), demonstrating progress in occlusion robustness.
• The use of synthesized image pairs from COCO and Pascal VOC improved STEm-Seg's AP by 2.4 points (from 13.8 to 16.2), showing data augmentation's effectiveness for occlusion generalization.
• Methods based on 3D convolutions and bottom-up architectures (e.g., STEm-Seg) showed stronger performance on occluded scenes, likely due to better temporal modeling and avoidance of detection bottlenecks.
• Failure cases revealed persistent issues such as ID switching, failure to re-identify fully occluded objects, and confusion in dense, overlapping scenes.
• The challenge results highlighted that even top methods struggle with severe occlusion, especially when objects are small, overlapping, or fully hidden, underscoring the need for improved contextual reasoning.