[Paper Review] A simple neural network module for relational reasoning

Adam Santoro, David Raposo+5 more|arXiv (Cornell University)|Jun 5, 2017

Multimodal Machine Learning ApplicationsComputer Science40 references501 citations

TL;DR

The paper introduces Relation Networks (RNs) as a plug-and-play module for relational reasoning, enabling CNN/LSTM architectures to perform state-of-the-art, super-human reasoning on CLEVR, bAbI, and dynamic physical systems tasks.

ABSTRACT

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

Motivation & Objective

• Motivate relational reasoning as central to intelligent behavior and identify limitations of standard neural networks in such tasks.
• Propose a simple, plug-and-play RN module to compute relations between object pairs.
• Demonstrate RN effectiveness across diverse domains: visual QA (CLEVR), text QA (bAbI), and dynamic physical systems.

Proposed method

• Define RN as RN(O)=f_phi(sum_{i,j} g_theta(o_i, o_j)) where o_i are object representations.
• Use g_theta to compute relations between object pairs and f_phi to aggregate relations.
• Allow inputs to be treated as a set of objects with order invariance via summation.
• Condition g_theta on auxiliary inputs like question embeddings when applicable.
• Show RN can operate on unstructured inputs by learning upstream object representations from CNN/LSTM features.
• Train end-to-end with Adam and standard CNN/LSTM components.

Experimental results

Research questions

• RQ1 Can a dedicated relational module improve neural networks’ ability to infer relations among objects in varied domains?
• RQ2 Do Relation Networks provide data-efficient, order-invariant relational reasoning when attached to existing architectures?
• RQ3 Are RNs capable of resolving relational questions in visual QA, text QA, and dynamic physical systems?

Key findings

• RN-augmented models achieve state-of-the-art, super-human performance on CLEVR from pixels (95.5% overall).
• RN-augmented models achieve 96.4% accuracy on CLEVR from state descriptions.
• On Sort-of-CLEVR, CNN+RN solves relational and non-relational questions with >94% accuracy, while CNN+MLP struggles on relational questions.
• On bAbI, the model solves 18/20 tasks with no catastrophic failures.
• In dynamic physical systems, RN infers connections with 93% accuracy and counts connected systems with 95% accuracy, outperforming an MLP.