[논문 리뷰] Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

Serhii Havrylov, Ivan Titov|arXiv (Cornell University)|2017. 05. 31.

Language and cultural evolution인용 수 153

한 줄 요약

The paper trains two neural agents to communicate via sequences of discrete symbols in a referential game, showing that straight-through Gumbel-softmax enables faster convergence and richer, compositional protocols, with exploration of natural-language grounding.

ABSTRACT

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.

연구 동기 및 목표

• Motivate learning to communicate from interaction rather than supervision.
• Demonstrate emergence of a language as sequences of discrete symbols in a referential game.
• Compare training methods (REINFORCE vs straight-through Gumbel-softmax) for efficiency and protocol quality.
• Investigate properties of the induced language, including compositionality and paraphrase-like variability.
• Explore indirect and direct grounding of emergent language in natural language.

제안 방법

• Agents are LSTMs (sender S and receiver R) operating on target images and a message m, produced as a token sequence from vocabulary V up to length L.
• Messages are discrete; gradients are estimated via REINFORCE or differentiable relaxations using Gumbel-softmax (GS) with straight-through (ST-GS) in training.
• GS-ST enables end-to-end differentiation by discretizing in the forward pass but using continuous relaxation in the backward pass.
• The loss encourages the receiver to identify the target image among distractors based on the message.
• Two grounding strategies are explored: indirect grounding via KL(qφ(m|t) || pω(m)) with a natural-language language model, and direct grounding via image captioning supervision.
• Temperature for Gumbel-softmax is learned per-step to stabilize training (τ(hs_i)) and is influenced by a learned inverse-temperature function.]
• research_questions:[
• Can two agents develop a meaningful, discrete-symbol communication protocol from scratch in a referential game?
• Is straight-through Gumbel-softmax faster and more effective than REINFORCE for learning discrete-language protocols?
• Does the emergent protocol exhibit compositionality and paraphrase-like variability akin to natural language?
• Does grounding emergent language in natural language (indirect or direct) improve interpretability or align with human language characteristics?

실험 결과

연구 질문

• RQ1 Can two agents develop a meaningful, discrete-symbol communication protocol from scratch in a referential game?
• RQ2 Is straight-through Gumbel-softmax faster and more effective than REINFORCE for learning discrete-language protocols?
• RQ3 Does the emergent protocol exhibit compositionality and paraphrase-like variability akin to natural language?
• RQ4 Does grounding emergent language in natural language (indirect or direct) improve interpretability or align with human language characteristics?

주요 결과

• Straight-through Gumbel-softmax converges faster than REINFORCE for learning symbol-sequence protocols in the referential game.
• Longer messages (higher L) aid faster convergence and yield more redundant (paraphrastic) encodings of the same content.
• The induced protocol shows hierarchical-like encoding and multiple paraphrases for the same semantic content.
• Grounding approaches (indirect KL regularization and optional captioning loss) can align emergent communication with natural language statistics and improve interpretability.
• Compared to natural-language grounding, the grounded protocol achieves similar communication success with differing omission scores, indicating partial alignment with content-word vs function-word distinctions.
• The ST-GS gradient direction behaves as a pseudogradient for this task, providing reliable optimization guidance.