[论文解读] Measuring abstract reasoning in neural networks

David G. T. Barrett, Felix Hill+3 more|arXiv (Cornell University)|Jul 11, 2018

Neural Networks and Applications被引用 143

一句话总结

这篇论文引入 Procedurally Generated Matrices (PGM) 数据集，用以探查神经网络的抽象视觉推理，结果表明专门的 Relational Network (WReN) 优于标准的 CNNs/ResNets，且辅助的符号解释提高了泛化能力。

ABSTRACT

Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.

研究动机与目标

• 推动并形式化使用 RPM 启发的矩阵，对神经网络的抽象视觉推理进行原则性的探针。
• 创建一个可控的、自动生成的 PGM 数据集，具备明确的抽象语义和多种泛化范式。
• 将标准架构与一种新颖的面向关系的网络进行比较，以识别推理所需的归纳偏置。
• 研究跨范式的泛化极限，并评估辅助的符号解释是否提升性能。

提出的方法

• 从定义好的关系、对象和属性三元组，程序化生成类似 RPM 的矩阵。
• 评估多种基线（CNN-MLP、ResNet 变体、LSTM）以及新颖的 Wild Relational Network (WReN)。
• 使用 Relation Network 核心来计算面板之间的关系并对候选答案进行评分。
• 使用 Adam 优化器进行训练，并在留出的验证集上进行超参数搜索。
• 尝试辅助的元目标训练，预测符号关系/对象/属性类型（二元元目标）。
• 比较在中性、内插、外推以及留出属性/三元组范式下的性能。

实验结果

研究问题

• RQ1 当有足够数据训练时，最先进的神经网络能否解决复杂的抽象推理任务？
• RQ2 模型在受控的范式转变下（内插、外推、留出的组件）对抽象推理的泛化能力如何？

主要发现

• CNNs 和标准 ResNets 在完整的 RPM 类推理任务上表现不佳。
• Wild Relational Network (WReN) 通过建模面板之间的成对关系，显著优于基线。
• 在内插和新组合范式中泛化最强，在外推和全新属性中较弱。
• 通过符号元目标的辅助训练，整体性能提升约 14%，并增强泛化性，尤其是对新颖组合。
• 元目标的预测确定性与任务准确度相关，表明对推理过程的解释有助于推理。