[Paper Review] GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Martin Engelcke, Adam R. Kosiorek+2 more|arXiv (Cornell University)|Jul 30, 2019

Generative Adversarial Networks and Image Synthesis51 references73 citations

TL;DR

GENESIS is an object-centric generative model for rendered 3D scenes that can both decompose scenes into objects and generate coherent novel scenes using an autoregressive prior over object components. It improves scene generation and decomposition over prior methods while enabling relational reasoning between components.

ABSTRACT

Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

Motivation & Objective

• Motivate learning compact, compositional representations of visual scenes to improve perception and planning in robotics and RL.
• Develop an unsupervised model that decomposes and generates scenes with explicit object-centric latent variables.
• Capture interactions between scene components with an autoregressive prior to enable coherent sampling of novel scenes.
• Provide scalable, parallelizable inference by performing components inference in low-dimensional latent space.

Proposed method

• Model a spatial Gaussian mixture over images where each component represents a scene element.
• Use an autoregressive prior over per-component masks encoded by an RNN to capture spatial relationships between components.
• Factorize component latents so that each z^c_k depends on z^m_k; render image as a sum of π_k p_θ(x|z^c_k).
• Train with amortized inference q_φ(z^m,z^c|x) that mirrors the generative structure; use a stick-breaking (SBP) prior for π_k or softmax normalization as an alternative.
• Apply Generalised ELBO with Constrained Optimisation (GECO) to balance reconstruction quality and KL regularization.
• Provide two variants: genesis with separate mask and component latents (z^m, z^c) and genesis-s with a single latent per component.

Experimental results

Research questions

• RQ1 Can an object-centric generative model decompose complex scenes and generate coherent novel scenes in an unsupervised fashion?
• RQ2 Does an autoregressive prior over scene components improve the coherence of generated scenes and enable relational reasoning between components?
• RQ3 How do object-centric representations learned by Genesis transfer to downstream tasks such as reasoning about scene state (e.g., stability) or predicting viewpoint?

Key findings

• Genesis achieves coherent, component-wise scene generation that respects spatial layout (e.g., floor/sky first, followed by objects, then background walls).
• On GQN, Genesis provides superior component-wise generation compared to MONet, producing semantically consistent scenes.
• Genesis achieves competitive or superior unsupervised segmentation metrics (ARI and segmentation-covering) versus MONet on ShapeStacks (ARI 0.73±0.03; SC 0.64±0.08; mSC 0.60±0.09).
• Representations learned by Genesis improve downstream task performance such as predicting tower stability and height, outperforming several baselines (e.g., bd-vae, dc-vae) on ShapeStacks tasks.
• Fréchet Inception Distances show Genesis variants reach best or competitive sample quality on Multi-dSprites and GQN (e.g., Multi-dSprites: 24.9/28.2; GQN: 80.5/70.2).