[Paper Review] Emergence of Grounded Compositional Language in Multi-Agent Populations

Igor Mordatch, Pieter Abbeel|arXiv (Cornell University)|Mar 15, 2017

Language and cultural evolution213 citations

TL;DR

The paper shows that grounded, compositional language can emerge among multi-agent populations in a physically grounded environment, learned end-to-end via differentiable multi-agent reinforcement learning with a discrete Gumbel-Softmax communication channel and vocabulary control.

ABSTRACT

By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing, including in machine translation, question answering, and sentiment analysis. However, for agents to intelligently interact with humans, simply capturing the statistical patterns is insufficient. In this paper we investigate if, and how, grounded compositional language can emerge as a means to achieve goals in multi-agent populations. Towards this end, we propose a multi-agent learning environment and learning methods that bring about emergence of a basic compositional language. This language is represented as streams of abstract discrete symbols uttered by agents over time, but nonetheless has a coherent structure that possesses a defined vocabulary and syntax. We also observe emergence of non-verbal communication such as pointing and guiding when language communication is unavailable.

Motivation & Objective

• Demonstrate emergence of a grounded, compositional language in a multi-agent, physically-situated setting.
• Show that agents develop a coherent vocabulary and syntax without human data or pre-defined meanings.
• Explore how environment variation and vocabulary constraints influence language structure and coordination.

Proposed method

• Use a cooperative partially observable Markov game with N agents and M landmarks in a continuous 2D space.
• Represent communication as discrete symbols emitted over time from a shared symbol vocabulary.
• Train identical policies for all agents end-to-end using backpropagation through time with differentiable dynamics.
• Employ a Gumbel-Softmax relaxation to enable differentiable sampling of discrete communication symbols.
• Incorporate an auxiliary goal prediction reward to encourage clear communicative signaling of goals.
• Penalize large vocabularies with a Dirichlet Process-based objective to encourage compositionality.

Experimental results

Research questions

• RQ1 Can grounded, compositional language emerge from interaction in a physical environment without human language data?
• RQ2 What environmental factors and vocabulary constraints promote compositional structure in the emergent language?
• RQ3 How does the emergent language relate to coordination strategies and task success across varying population sizes and configurations?
• RQ4 What non-verbal communication strategies accompany or substitute language when communication channels are limited?

Key findings

• A compositional, interpretable symbol vocabulary emerges that maps to landmarks and actions, with structure shaped by the environment.
• Vocabulary size penalties and diverse task configurations promote the emergence of compositional language rather than a holistic, non-compositional one.
• Agents achieve substantially higher task rewards with communication than without (training: -0.919 vs -0.332; test: -0.920 vs -0.392).
• Emergent language reflects grounding in physics, with utterances often preceding actions (e.g., GOTO uttered before movement).
• Non-verbal strategies (pointing, gaze-based signaling, and even pushing) arise when language is unavailable, illustrating multi-modal coordination.
• The system generalizes to unseen configurations, such as different numbers of landmarks or agents, by reusing compositional concepts.