[Paper Review] Learning a Spatio-Temporal Embedding for Video Instance Segmentation

Anthony Hu, Alex Kendall+1 more|arXiv (Cornell University)|Dec 19, 2019

Video Surveillance and Tracking Methods29 references18 citations

TL;DR

This paper proposes a real-time, causal spatio-temporal embedding network for video instance segmentation that jointly leverages appearance, motion, and monocular depth to cluster pixels of the same object across time. By using a 3D causal convolutional network and self-supervised depth estimation, the method achieves state-of-the-art performance on the KITTI MOTS dataset, enabling robust tracking through occlusions and missed detections without post-processing.

ABSTRACT

We present a novel embedding approach for video instance segmentation. Our method learns a spatio-temporal embedding integrating cues from appearance, motion, and geometry; a 3D causal convolutional network models motion, and a monocular self-supervised depth loss models geometry. In this embedding space, video-pixels of the same instance are clustered together while being separated from other instances, to naturally track instances over time without any complex post-processing. Our network runs in real-time as our architecture is entirely causal - we do not incorporate information from future frames, contrary to previous methods. We show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.

Motivation & Objective

• To develop a real-time video instance segmentation method that maintains temporal consistency without relying on future frame information.
• To improve instance tracking robustness under occlusions and missing detections by leveraging spatio-temporal embeddings.
• To integrate monocular depth estimation as a self-supervised supervisory signal to enhance embedding disambiguation.
• To eliminate complex post-processing by directly learning a clustering-friendly embedding space through end-to-end training.
• To advance the state-of-the-art on the KITTI Multi-Object and Tracking Dataset (MOTS) using a fully causal architecture.

Proposed method

• A 3D causal convolutional neural network processes past frames to encode spatio-temporal motion cues in the embedding space.
• A self-supervised monocular depth loss is used to predict depth from consecutive frames, improving geometric consistency in the embedding.
• The spatio-temporal embedding loss encourages pixels of the same instance to cluster closely while remaining separated from other instances.
• Instance correspondence is established via mean shift clustering on accumulated past embeddings, using attraction and repulsion radii to match instances over time.
• The model is trained end-to-end with a joint loss combining instance-level embedding consistency and depth prediction.
• All components are designed to be causal, ensuring real-time inference by only using past and current frames.

Experimental results

Research questions

• RQ1 Can a causal 3D convolutional network effectively model motion and temporal context for video instance segmentation?
• RQ2 Does incorporating self-supervised monocular depth improve the quality and robustness of spatio-temporal embeddings?
• RQ3 Can the proposed embedding loss enable consistent instance tracking through occlusions and missing detections without post-processing?
• RQ4 How does sequence length affect the stability and performance of the learned spatio-temporal embeddings?
• RQ5 To what extent does the integration of geometry enhance disambiguation between similar-looking or overlapping objects?

Key findings

• The model achieves a MOTSA score of 0.714 and sMOTSA of 0.644 on the KITTI MOTS dataset, advancing the state-of-the-art.
• Including ground-truth clustering and background segmentation improves sMOTSA from 0.461 to 0.644, demonstrating the benefit of accurate clustering.
• The optimal sequence length for training is five frames, as longer sequences degrade performance due to conflicting long-range embedding constraints.
• Performance drops significantly on sequences longer than 10 frames (sMOTSA: 0.273), indicating limitations in modeling long-term dynamics.
• The integration of depth improves embedding structure and enables correct tracking during total and partial occlusions, as shown in qualitative results.
• The model successfully tracks instances through full occlusion, such as the green car in Figure 3(c), due to preserved temporal context in the embedding space.