[Paper Review] Benchmarking Unsupervised Object Representations for Video Sequences

Marissa A. Weis, Kashyap Chitta+5 more|arXiv (Cornell University)|Jun 12, 2020

Domain Adaptation and Few-Shot Learning43 references4 citations

TL;DR

This paper introduces a benchmark for evaluating unsupervised object-centric video representation models using procedurally generated video datasets with increasing complexity and challenging tracking scenarios. It compares four methods—ViMON, OP3, TBA, and SCALOR—finding that models with unconstrained latent spaces outperform spatial transformer-based architectures in detection, segmentation, and tracking, despite all struggling with occlusion and complex dynamics.

ABSTRACT

Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding. Recently, several methods have been proposed for unsupervised learning of object-centric representations. However, since these models were evaluated on different downstream tasks, it remains unclear how they compare in terms of basic perceptual abilities such as detection, figure-ground segmentation and tracking of objects. To close this gap, we design a benchmark with four data sets of varying complexity and seven additional test sets featuring challenging tracking scenarios relevant for natural videos. Using this benchmark, we compare the perceptual abilities of four object-centric approaches: ViMON, a video-extension of MONet, based on recurrent spatial attention, OP3, which exploits clustering via spatial mixture models, as well as TBA and SCALOR, which use explicit factorization via spatial transformers. Our results suggest that the architectures with unconstrained latent representations learn more powerful representations in terms of object detection, segmentation and tracking than the spatial transformer based architectures. We also observe that none of the methods are able to gracefully handle the most challenging tracking scenarios despite their synthetic nature, suggesting that our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.

Motivation & Objective

• To address the lack of standardized evaluation for unsupervised object-centric video representation models across different architectures.
• To assess the perceptual abilities of object-centric models—specifically detection, segmentation, and tracking—using a common benchmark independent of downstream tasks.
• To identify architectural inductive biases that influence performance in object-centric video representation learning.
• To evaluate model generalization under challenging tracking scenarios such as occlusion, re-entry, and object merging.

Proposed method

• The benchmark consists of four procedurally generated video datasets (VMDS, SpMOT, VOR, texVMDS) with increasing visual complexity and two generalization test sets.
• The evaluation protocol uses the multi-object tracking (MOT) framework to quantitatively measure detection, segmentation, and tracking performance via metrics like MOTA and IDF1.
• Four object-centric models are evaluated: ViMON (recurrent spatial attention), OP3 (spatial mixture modeling with clustering), TBA and SCALOR (spatial transformer-based factorization).
• Each model is trained and tested under identical conditions using the same video sequences and evaluation protocol to ensure fair comparison.
• The benchmark includes challenging scenarios such as object occlusion, re-entry after occlusion, and object merging, simulating realistic video dynamics.
• Results are reported using standard MOT metrics, including precision, recall, and identity F1 score, with IoU thresholds for matching ground truth and predicted objects.

Experimental results

Research questions

• RQ1 How do different object-centric video representation models compare in terms of object detection, segmentation, and tracking under a common evaluation protocol?
• RQ2 Which architectural choices—unconstrained latent spaces versus explicit spatial transformers—lead to better perceptual performance in unsupervised video representation learning?
• RQ3 To what extent do models generalize to challenging tracking scenarios such as occlusion, object merging, and re-entry?
• RQ4 How do inductive biases in model design, such as clustering or spatial attention, affect robustness and accuracy in object-centric video understanding?

Key findings

• Models with unconstrained latent representations, such as ViMON, outperform spatial transformer-based models (TBA and SCALOR) in object detection, segmentation, and tracking across all benchmark datasets.
• OP3 achieves the highest quantitative performance in terms of MOTA and IDF1 scores but shows strong dependency on color cues and accumulates false positives when the number of objects is fewer than the number of slots.
• Despite explicit depth reasoning and amodal mask prediction, TBA and SCALOR perform worse than ViMON, suggesting that their architectural mechanisms may not yet be effectively leveraging these inductive biases.
• All models struggle significantly with occlusion, even in synthetic settings, indicating a fundamental gap in robustness to dynamic visual challenges.
• The benchmark reveals that current unsupervised object-centric models fail to generalize gracefully to complex tracking dynamics, highlighting the need for more robust architectural designs.
• The public benchmark, code, and leaderboard are released to support future comparative research in unsupervised video representation learning.