[Paper Review] Self-supervised Amodal Video Object Segmentation

Jian Yao, Yuxin Hong+7 more|arXiv (Cornell University)|Oct 23, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes SaVos, a self-supervised video object segmentation framework that infers amodal (complete) object masks by leveraging visible object parts across video frames, using spatiotemporal priors to explain away occluded regions without requiring explicit amodal annotations. It achieves state-of-the-art performance on synthetic (FISHBOWL) and real-world (KINS-Video-Car) benchmarks and shows strong generalization via test-time adaptation, outperforming models trained on target-domain data.

ABSTRACT

Amodal perception requires inferring the full shape of an object that is partially occluded. This task is particularly challenging on two levels: (1) it requires more information than what is contained in the instant retina or imaging sensor, (2) it is difficult to obtain enough well-annotated amodal labels for supervision. To this end, this paper develops a new framework of Self-supervised amodal Video object segmentation (SaVos). Our method efficiently leverages the visual information of video temporal sequences to infer the amodal mask of objects. The key intuition is that the occluded part of an object can be explained away if that part is visible in other frames, possibly deformed as long as the deformation can be reasonably learned. Accordingly, we derive a novel self-supervised learning paradigm that efficiently utilizes the visible object parts as the supervision to guide the training on videos. In addition to learning type prior to complete masks for known types, SaVos also learns the spatiotemporal prior, which is also useful for the amodal task and could generalize to unseen types. The proposed framework achieves the state-of-the-art performance on the synthetic amodal segmentation benchmark FISHBOWL and the real world benchmark KINS-Video-Car. Further, it lends itself well to being transferred to novel distributions using test-time adaptation, outperforming existing models even after the transfer to a new distribution.

Motivation & Objective

• To address the challenge of amodal video object segmentation without relying on costly amodal annotations.
• To leverage temporal video sequences to infer occluded object parts using visible parts from other frames as self-supervision.
• To improve generalization to unseen object types and distribution shifts through spatiotemporal priors and test-time adaptation.
• To develop a model that is inherently compatible with test-time adaptation, enabling performance boosts on new distributions without retraining.

Proposed method

• The framework learns dense amodal object motion across frames to bridge visible and occluded regions, using temporal consistency as a supervisory signal.
• It jointly predicts the amodal mask and dense motion on the amodal mask, avoiding explicit pixel correspondence by exploiting spatiotemporal structure.
• A novel architecture design incorporates inductive bias from Gestalt psychology’s common-fate principle to improve motion and mask prediction on occluded areas.
• Self-supervision is derived from visible masks in other frames, with a surrogate loss optimizing only visible parts to avoid overfitting to intrusive predictions.
• Test-time adaptation is applied without modifying the self-supervised loss, using early stopping to prevent over-completion and maintain generalization.
• The method learns both spatiotemporal priors and implicit type priors, with the latter suppressed during adaptation to enhance robustness to unseen categories.

Experimental results

Research questions

• RQ1 Can visible object parts in other video frames be effectively used to self-supervise amodal mask prediction without explicit amodal annotations?
• RQ2 How can spatiotemporal priors be leveraged to improve amodal segmentation beyond type-based priors?
• RQ3 Does the proposed self-supervised framework generalize better to unseen object types and distribution shifts than image-level baselines?
• RQ4 Why is test-time adaptation particularly effective for SaVos compared to other self-supervised methods?

Key findings

• SaVos achieves state-of-the-art performance on the synthetic FISHBOWL benchmark and the real-world KINS-Video-Car dataset in a self-supervised setting.
• On KINS-Video-Car, SaVos trained on synthetic fish data outperforms a model fine-tuned on the real-world driving car dataset after test-time adaptation.
• Test-time adaptation significantly improves generalization, with SaVos showing a larger performance boost than image-level baselines like Self-Deocclusion.
• The model learns type priors implicitly, but early stopping during test-time adaptation helps suppress over-completion and intrusive predictions on unseen types.
• The self-supervised loss design enables efficient test-time adaptation, as the training signal is directly aligned with occluded regions in the test video via amodal motion modeling.
• Empirical results show that SaVos is more adaptation-friendly than alternative self-supervised methods, indicating its loss design is better suited for distribution shift generalization.