[论文解读] Artificial Agents Learn Flexible Visual Representations by Playing a Hiding Game.

Luca Weihs, Aniruddha Kembhavi+6 more|arXiv (Cornell University)|Dec 17, 2019

Reinforcement Learning in Robotics参考文献 59被引用 7

一句话总结

该论文表明，在高保真、交互式的环境中，人工智能代理通过玩藏宝游戏（cache）学习到灵活的视觉表征，能够编码遮挡、物体恒存性、自由空间和包含关系——其性能与最先进的监督式表征学习方法相当，但无需大规模标注数据集，而是通过具身化、交互式的游戏实现。

ABSTRACT

The ubiquity of embodied gameplay, observed in a wide variety of animal species including turtles and ravens, has led researchers to question what advantages play provides to the animals engaged in it. Mounting evidence suggests that play is critical in developing the neural flexibility for creative problem solving, socialization, and can improve the plasticity of the medial prefrontal cortex. Comparatively little is known regarding the impact of gameplay upon embodied artificial agents. While recent work has produced artificial agents proficient in abstract games, the environments these agents act within are far removed the real world and thus these agents provide little insight into the advantages of embodied play. Hiding games have arisen in multiple cultures and species, and provide a rich ground for studying the impact of embodied gameplay on representation learning in the context of perspective taking, secret keeping, and false belief understanding. Here we are the first to show that embodied adversarial reinforcement learning agents playing cache, a variant of hide-and-seek, in a high fidelity, interactive, environment, learn representations of their observations encoding information such as occlusion, object permanence, free space, and containment; on par with representations learnt by the most popular modern paradigm for visual representation learning which requires large datasets independently labeled for each new task. Our representations are enhanced by intent and memory, through interaction and play, moving closer to biologically motivated learning strategies. These results serve as a model for studying how facets of vision and perspective taking develop through play, provide an experimental framework for assessing what is learned by artificial agents, and suggest that representation learning should move from static datasets and towards experiential, interactive, learning.

研究动机与目标

• 探究具身化、交互式游戏是否能驱动人工智能代理发展灵活的视觉表征。
• 探讨在真实、交互式环境中进行游戏如何促进视角转换、错误信念和物体恒存性等认知概念的出现。
• 将通过游戏学习到的表征质量与传统大规模监督式表征学习范式进行比较。
• 开发一种实验框架，用于评估人工智能代理通过经验性、交互式学习所获得的知识。

提出的方法

• 代理在高保真、交互式3D环境中通过对抗性强化学习进行训练，其中一方代理负责藏匿物体，另一方负责搜寻。
• 该环境支持丰富的物理交互，包括物体遮挡、包含关系以及动态场景变化，从而实现逼真的游戏动态。
• 从代理内部神经激活中提取视觉表征，并评估其编码空间与感知概念（如自由空间、物体恒存性）的能力。
• 通过标准下游任务，将表征质量与最先进的自监督和监督式视觉表征学习方法进行基准对比。
• 通过引入记忆建模与意图建模，增强学习过程，使代理能够追踪隐藏物体状态并进行战略性规划。
• 该框架支持分析交互与游戏如何随时间塑造视觉表征的结构。

实验结果

研究问题

• RQ1 人工智能代理是否能在真实环境中通过具身化、交互式游戏发展出灵活的视觉表征？
• RQ2 通过游戏学习到的表征在多大程度上编码了遮挡、物体恒存性、包含关系等核心感知与认知概念？
• RQ3 通过游戏学习到的表征质量与大规模监督式表征学习方法相比如何？
• RQ4 记忆与意图建模在提升所学表征的质量与灵活性方面起到何种作用？

主要发现

• 玩藏宝游戏的人工智能代理学习到的视觉表征能够以高保真度编码遮挡、物体恒存性、自由空间和包含关系。
• 尽管代理在无外部监督或大规模标注数据集的情况下进行学习，这些表征的质量仍与最先进的监督式表征学习方法相当。
• 通过游戏进行的表征学习在下游任务上的表现优于或等同于标准的自监督预训练基线方法。
• 记忆与意图建模的整合显著提升了代理对隐藏状态的推理能力以及游戏中的有效规划能力。
• 结果表明，交互式、经验性学习可产生丰富且符合生物学原理的表征，与数据密集型、静态学习范式所获得的表征相当。
• 该框架为研究人工智能代理通过游戏发展视觉能力与视角转换能力提供了可行的实验模型。