[Paper Review] SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition

Zhixuan Lin, Yifu Wu+6 more|arXiv (Cornell University)|Jan 8, 2020

Advanced Image and Video Retrieval TechniquesComputer Science25 references44 citations

TL;DR

SPACE combines spatial-attention and scene-mixture in a probabilistic model to jointly decompose foreground objects and complex backgrounds with parallel foreground processing for scalable unsupervised object-centric scene representation, evaluated on Atari and 3D-Rooms against SPAIR, IODINE, and GENESIS.

ABSTRACT

The ability to decompose complex multi-object scenes into meaningful abstractions like objects is fundamental to achieve higher-level cognition. Previous approaches for unsupervised object-oriented scene representation learning are either based on spatial-attention or scene-mixture approaches and limited in scalability which is a main obstacle towards modeling real-world scenes. In this paper, we propose a generative latent variable model, called SPACE, that provides a unified probabilistic modeling framework that combines the best of spatial-attention and scene-mixture approaches. SPACE can explicitly provide factorized object representations for foreground objects while also decomposing background segments of complex morphology. Previous models are good at either of these, but not both. SPACE also resolves the scalability problems of previous methods by incorporating parallel spatial-attention and thus is applicable to scenes with a large number of objects without performance degradations. We show through experiments on Atari and 3D-Rooms that SPACE achieves the above properties consistently in comparison to SPAIR, IODINE, and GENESIS. Results of our experiments can be found on our project website: https://sites.google.com/view/space-project-page

Motivation & Objective

• Motivate unsupervised learning of structured scene representations for multi-object scenes with occlusion and complex backgrounds.
• Propose SPACE to unify spatial-attention and scene-mixture approaches within a probabilistic latent-variable framework.
• Enable parallel processing of foreground objects to address scalability while preserving disentangled object representations.

Proposed method

• Introduce a foreground module with parallel spatial-attention to generate z_where, z_0pt, z_pres, and z_what per grid cell.
• Use a Spatial Transformer to render each foreground object in parallel onto a canvas.
• Model background with a K-component pixel-wise mixture, each with latent z^m (mixing) and z^c (color) decoded by a VAE.
• Train with a variational objective (ELBO) that jointly considers foreground and background, with a mean-field approximation for cell latents.
• Prevent box-splitting via an auxiliary boundary loss that discourages object masks from touching the glimpse boundary.
• Demonstrate scalability by parallel foreground processing, contrasting with sequential inference in SPAIR, IODINE, and GENESIS.

Experimental results

Research questions

• RQ1 Can SPACE provide explicit object-centric foreground representations while also decomposing complex background components?
• RQ2 Does parallel foreground processing improve scalability and speed without sacrificing foreground detection quality?
• RQ3 How does SPACE compare to SPAIR, IODINE, and GENESIS in terms of convergence, speed, and bounding-box quality on Atari and 3D-Room datasets?

Key findings

• SPACE achieves comparable bounding-box quality to SPAIR while offering orders-of-magnitude faster gradient-step latency and training convergence.
• SPACE scales to large numbers of foreground objects without significant performance degradation due to parallel foreground processing.
• SPACE provides explicit, disentangled foreground objects with per-object properties (position, scale) and decomposed background components, outperforming baselines on 3D-Room and Atari in qualitative analyses.
• Quantitative results show SPACE competitive with SPAIR in average precision and lower object-count error rates on 3D-Room Large settings, with faster convergence and parallelized rendering.
• Background: SPACE’s background is decomposed into multiple components, enabling better modeling of complex morphologies than models that treat background as a single blob.