[Paper Review] Grounded Language Learning in a Simulated 3D World

Karl Moritz Hermann, Felix Hill+12 more|arXiv (Cornell University)|Jun 20, 2017

Natural Language Processing TechniquesComputer Science27 references153 citations

TL;DR

This paper presents an agent that learns to ground natural language in a 3D simulated world by combining reinforcement learning with unsupervised auxiliary objectives, enabling zero-shot comprehension and generalization to novel instructions. The approach integrates visual perception, language processing, and action policies end-to-end and demonstrates curriculum-enabled multi-task learning and semantic bootstrapping.

ABSTRACT

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.

Motivation & Objective

• Motivate grounding language learning as a means for scalable human–AI interaction in continuous, embodied environments.
• Develop an end-to-end agent that maps linguistic expressions to perceptual representations and actions using pixel-level input.
• Show that combining reinforcement learning with unsupervised auxiliary tasks accelerates learning and enables generalization to novel commands.
• Demonstrate curriculum and multi-task learning to acquire and transfer semantic knowledge across tasks and environments.

Proposed method

• Agent composed of four interconnected neural modules: visual encoder (V), language encoder (L), a mixing module (M), and an action/policy module (A).
• Training uses Advantage Actor-Critic with asynchronous 32 threads and RMSProp optimization.
• Auxiliary unsupervised objectives include temporal autoencoding (tAE) to predict next visual input and a language prediction (LP) task to predict instruction words from observations.
• Additional auxiliary tasks experimented with include reward prediction (RP) and value replay (VR) to stabilize reinforcement learning.
• Learning from rewards is supplemented by predictions about the world to shape representation learning and policy optimization.

Experimental results

Research questions

• RQ1 Can an agent learn grounded meanings for linguistic expressions from raw pixel input in a continuous 3D environment?
• RQ2 Does combining reinforcement learning with unsupervised auxiliary objectives enable efficient word learning and generalization to novel instructions?
• RQ3 Can an agent decompose and compose lexical concepts to interpret unfamiliar phrases and extend relational language to new objects?
• RQ4 Does curriculum learning enable multi-task grounding of language tied to actions and relations across tasks?

Key findings

• Reinforcement learning alone yields little learning; auxiliary objectives (tAE, LP, RP, VR) substantially enable vocabulary acquisition.
• Word learning speed improves when the agent has prior lexical knowledge, indicating bootstrap of semantic knowledge accelerates new word acquisition.
• Agents generalize to unseen words and novel combinations through decomposition and productive composition of known concepts.
• Curriculum learning enables solving progressively more complex referring expressions and multi-task language grounding.
• A single agent can learn multiple tasks (Selection, Next to, In room) via a two-step curriculum, showing transfer of language-grounded policies to larger environments.