[Paper Review] Discovering objects and their relations from entangled scene representations

David Raposo, Adam Santoro+4 more|arXiv (Cornell University)|Feb 16, 2017

Multimodal Machine Learning Applications73 citations

TL;DR

Relation Networks (RNs) learn object relations in scenes, are permutation-invariant, and can factorize entangled inputs; they can be paired with memory-augmented nets for one-shot relation learning.

ABSTRACT

Our world can be succinctly and compactly described as structured scenes of objects and relations. A typical room, for example, contains salient objects such as tables, chairs and books, and these objects typically relate to each other by their underlying causes and semantics. This gives rise to correlated features, such as position, function and shape. Humans exploit knowledge of objects and their relations for learning a wide spectrum of tasks, and more generally when learning the structure underlying observed data. In this work, we introduce relation networks (RNs) - a general purpose neural network architecture for object-relation reasoning. We show that RNs are capable of learning object relations from scene description data. Furthermore, we show that RNs can act as a bottleneck that induces the factorization of objects from entangled scene description inputs, and from distributed deep representations of scene images provided by a variational autoencoder. The model can also be used in conjunction with differentiable memory mechanisms for implicit relation discovery in one-shot learning tasks. Our results suggest that relation networks are a potentially powerful architecture for solving a variety of problems that require object relation reasoning.

Motivation & Objective

• Motivate the need for reasoning about objects and relations in structured scenes.
• Propose a neural architecture (Relation Networks) that operates on pairs of objects with permutation invariance.
• Demonstrate RN capability to classify scenes based on relational structure.
• Show RN as a bottleneck to factorize entangled scene inputs into object-like representations.
• Demonstrate combination of RN with memory-augmented networks for one-shot relation learning.

Proposed method

• Define objects as feature vectors in a scene description matrix D (m objects by n features).
• Compute relations with a shared MLP gψ on all pairs of objects and aggregate via a commutative/associative function a (typically sum).
• Use a final function fφ to produce predictions from a, i.e., r̃ = fφ(Σij gψ(oi, oj)).
• Evaluate RN on supervised tasks where targets are adjacency matrices describing object relations.
• Demonstrate RN can induce factorization of objects from entangled inputs using a linear bottleneck layer or a VAE preprocessor.
• Combine RN with a Memory-Augmented Neural Network (MANN) to perform one-shot relation learning.

Experimental results

Research questions

• RQ1 Can RNs learn and generalize object-relational structure from scene descriptions?
• RQ2 Can RNs infer object factorization and relations from entangled or pixel-based inputs?
• RQ3 Do RNs support one-shot learning when combined with memory modules?
• RQ4 How does RN performance compare to MLP baselines on relational tasks?
• RQ5 Can RN-mediated representations enable generalization to unseen relational graphs?

Key findings

• RNs outperform similarly sized MLPs on relational scene classification tasks and generalize to unseen classes.
• RNs can infer object relations from entangled inputs by learning a linear disentangler before the RN, revealing emergent object-factorized representations.
• A VAE-based perceptual pathway can feed latent codes into the RN, demonstrating RN compatibility with distributed image representations.
• RN-preprocessed MANNs achieve high one-shot relational classification within episodes, while MANNs with MLP preprocessors perform at chance.
• RN capabilities persist when used with memory and perceptual modules, indicating broad applicability for relational reasoning tasks.