[论文解读] Emergent Systematic Generalization In a Situated Agent

Felix Hill, Andrew K. Lampinen+5 more|arXiv (Cornell University)|Oct 1, 2019

Multimodal Machine Learning Applications被引用 22

一句话总结

本文在3D模拟环境中研究了情境智能体的系统性泛化，表明当智能体使用多样化、多模态的观察数据进行训练时，其在分布外指令上的表现显著提升。关键因素包括高频率的对象/词汇接触、基于视角的视觉不变性以及感知输入的多样性，表明神经网络在具备丰富、多样的感官体验（类似人类学习）时，泛化能力更强。

ABSTRACT

The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.

研究动机与目标

• 探究深度神经网络是否能在情境化3D环境中对从未见过的指令实现系统性泛化。
• 识别影响视觉-语言智能体泛化性能的具体训练与环境因素。
• 探讨多模态、感知丰富的观察输入如何促进神经网络中系统性泛化的出现。

提出的方法

• 在3D Unity模拟环境中训练一种通用智能体架构，使其基于自然语言指令执行物体操作任务。
• 通过改变对象/词汇接触对的数量来调整训练制度，以评估其对泛化性能的影响。
• 通过操控智能体的视觉视角和参考框架，评估视觉不变性对性能的影响。
• 通过改变视觉输入（如物体位置、光照条件和视角）来增加感知多样性，以测试其对学习的影响。

实验结果

研究问题

• RQ1 在训练过程中，对象/词汇接触次数的增加如何影响智能体对未见指令的泛化能力？
• RQ2 智能体的参考框架或视觉视角在多大程度上影响系统性泛化？
• RQ3 视觉输入的感知可变性在多大程度上影响智能体超越训练分布的泛化能力？

主要发现

• 在训练集中增加对象/词汇接触次数，显著提升了智能体对未见指令的泛化能力。
• 由智能体视角引入的视觉不变性在实现系统性泛化中起到了关键作用。
• 视觉输入中更高的感知多样性带来了更强的泛化性能，表明更丰富的感官输入可增强学习效果。
• 结果表明，神经网络的系统性泛化高度依赖于环境与训练特定的设计选择。