[论文解读] Grounded Language Learning in a Simulated 3D World

Karl Moritz Hermann, Felix Hill+12 more|arXiv (Cornell University)|Jun 20, 2017

Natural Language Processing Techniques参考文献 27被引用 153

一句话总结

这篇论文提出一个智能体，通过将强化学习与无监督辅助目标相结合，在三维仿真世界中学习将自然语言 grounding（对齐/定位）到感知，能够实现零样本理解和对新指令的泛化。该方法将视觉感知、语言处理和行动策略端到端整合，并展示了具备课程驱动的多任务学习与语义自举能力。

ABSTRACT

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.

研究动机与目标

• 将语言 grounding 学习作为在连续、具身环境中实现可扩展的人机交互的手段。
• 开发一个端到端的智能体，使用像素级输入将语言表达映射到感知表示和行动。
• 证明将强化学习与无监督辅助任务相结合能够加速学习并实现对新指令的泛化。
• 展示课程学习和多任务学习，以在不同任务和环境中获得并迁移语义知识。

提出的方法

• 由四个互相关联的神经网络模块组成的智能体：视觉编码器（V）、语言编码器（L）、混合模块（M）以及动作/策略模块（A）。
• 训练采用异步32线程的 Advantage Actor-Critic 与 RMSProp 优化。
• 辅助无监督目标包括时间自编码（tAE）以预测下一个视觉输入，以及语言预测（LP）任务以从观测中预测指令单词。
• 还尝试的其他辅助任务包括奖励预测（RP）和值回放（VR），以稳定强化学习。
• 通过对世界的预测来辅助学习，以塑造表征学习和策略优化，并结合对奖励的学习。

实验结果

研究问题

• RQ1 代理人是否能够从原始像素输入在连续三维环境中学习到语言表达的 Grounded 含义？
• RQ2 将强化学习与无监督辅助目标结合是否能够实现高效的词汇学习并对新指令进行泛化？
• RQ3 代理人是否能够分解并组合词汇概念来解释不熟悉的短语，并将关系语言扩展到新对象？
• RQ4 课程学习是否能够实现将语言 grounding 与行动和关系跨任务的多任务学习？

主要发现

• 单独的强化学习几乎没有学习收益；辅助目标（tAE、LP、RP、VR）显著促进词汇习得。
• 当智能体具备前置词汇知识时，单词学习速度有所提升，表明语义知识的自举有助于新单词的获取。
• 智能体通过已知概念的分解与产出式组合，能够泛化到未见过的单词和新组合。
• 课程学习使得对指称表达的逐步复杂化和多任务语言 grounding 成为可能。
• 单个智能体可以通过两步课程学习来同时掌握多任务（Selection、Next to、In room），并展示语言 grounding 策略向更大环境的迁移。