[Paper Review] Measuring abstract reasoning in neural networks

David G. T. Barrett, Felix Hill+3 more|arXiv (Cornell University)|Jul 11, 2018

Neural Networks and ApplicationsComputer Science143 citations

TL;DR

This paper introduces the Procedurally Generated Matrices (PGM) dataset to probe abstract visual reasoning in neural networks, showing that a specialized Relational Network (WReN) outperforms standard CNNs/ResNets and that auxiliary symbolic explanations improve generalisation.

ABSTRACT

Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.

Motivation & Objective

• Motivate and formalise a principled probe of abstract visual reasoning in neural nets using RPM-inspired matrices.
• Create a controllable, automatically generated PGM dataset with explicit abstract semantics and multiple generalisation regimes.
• Compare standard architectures to a novel relation-focused network to identify necessary inductive biases for reasoning.
• Investigate generalisation limits across regimes and assess whether auxiliary symbolic explanations improve performance.

Proposed method

• Procedurally generate RPM-like matrices from defined relation, object, and attribute triples.
• Evaluate multiple baselines (CNN-MLP, ResNet variants, LSTM) and a novel Wild Relational Network (WReN).
• Use a Relation Network core to compute inter-panel relations and score candidate answers.
• Train with Adam optimizer and perform hyper-parameter sweeps on a held-out validation set.
• Experiment with auxiliary meta-target training that predicts symbolic relation/object/attribute types (binary meta-targets).
• Compare performance across neutral, interpolation, extrapolation, and held-out attribute/triple regimes.

Experimental results

Research questions

• RQ1 Can state-of-the-art neural networks solve complex abstract reasoning tasks when trained with sufficient data?
• RQ2 How well do models generalise abstract reasoning across controlled regime shifts (interpolation, extrapolation, held-out components)?

Key findings

• CNNs and standard ResNets perform poorly on full RPM-like reasoning tasks.
• The Wild Relational Network (WReN) substantially outperforms baselines by modeling pairwise relations between panels.
• Generalisation is strongest in interpolation and in novel-combination regimes, weaker for extrapolation and entirely new attributes.
• Auxiliary training with symbolic meta-targets improves overall performance by about 14% and enhances generalisation, especially for novel combinations.
• Prediction certainty of meta-targets correlates with task accuracy, indicating usefulness of explanations for reasoning.”