[Paper Review] MONet: Unsupervised Scene Decomposition and Representation

Christopher Burgess, Löıc Matthey+5 more|arXiv (Cornell University)|Jan 22, 2019

Advanced Neural Network ApplicationsComputer Science32 references194 citations

TL;DR

MONet is an unsupervised model that decomposes scenes into object-like components using a recurrent attention network and a shared VAE, learning disentangled representations and handling occlusion across 3D and 2D datasets.

ABSTRACT

The ability to decompose scenes in terms of abstract building blocks is crucial for general intelligence. Where those basic building blocks share meaningful properties, interactions and other regularities across scenes, such decompositions can simplify reasoning and facilitate imagination of novel scenarios. In particular, representing perceptual observations in terms of entities should improve data efficiency and transfer performance on a wide range of tasks. Thus we need models capable of discovering useful decompositions of scenes by identifying units with such regularities and representing them in a common format. To address this problem, we have developed the Multi-Object Network (MONet). In this model, a VAE is trained end-to-end together with a recurrent attention network -- in a purely unsupervised manner -- to provide attention masks around, and reconstructions of, regions of images. We show that this model is capable of learning to decompose and represent challenging 3D scenes into semantically meaningful components, such as objects and background elements.

Motivation & Objective

• Motivate learning scene representations as decomposable objects to improve reasoning and data efficiency.
• Develop an unsupervised architecture that segments scenes into multiple components with a common latent space.
• Enable handling of occlusion and varying numbers of objects without labeled segmentations.
• Demonstrate generalization to scenes with more/fewer objects and novel configurations.
• Show that learned components yield disentangled, interpretable latent factors.

Proposed method

• Use a recurrent attention network to generate a sequence of masks that cover the scene without supervision.
• Model each masked region with a component VAE that reconstructs only the masked pixels, allowing occluded areas to be inferred.
• Train end-to-end with a loss combining VAE reconstruction, KL regularization on per-slot latents, and a KL term aligning attention masks with decoded masks.
• Maintain a scope variable to ensure the masks cover the whole image across K slots (sum of masks equals one).
• Allow a variable number of slots (K) and generalize at test time to scenes with more or fewer objects.
• Optimize with hyperparameters beta and gamma controlling disentanglement and mask modeling respectively.

Experimental results

Research questions

• RQ1 Can MONet learn to decompose complex scenes into semantically meaningful objects without supervision?
• RQ2 Do learned masks correspond to meaningful scene elements like objects, walls, and background?
• RQ3 Can MONet handle occlusion and varying object counts, and generalize to unseen configurations?
• RQ4 Do the latent representations across slots disentangle into interpretable features?
• RQ5 How does compositional processing affect reconstruction efficiency and accuracy?

Key findings

• MONet achieves unsupervised decomposition of non-trivial 3D scenes into objects and background elements.
• The model generalizes to additional slots at test time (e.g., 9 slots) and to scenes with more objects than seen during training.
• Latent factors within slots show disentanglement with interpretable features controllable via traversals.
• MONet accurately segments and reconstructs scenes with occluded objects and highly overlapping shapes across datasets (Objects Room, Multi-dSprites, CLEVR).
• The approach yields coherent inpainting of occluded regions and demonstrates robust cross-dataset applicability.