[Paper Review] On the interaction between supervision and self-play in emergent communication

Ryan Lowe, Abhinav Gupta+3 more|arXiv (Cornell University)|Feb 4, 2020

Language and cultural evolution31 references30 citations

TL;DR

The paper studies how to combine supervised learning with self-play (S2P) for emergent communication, showing that supervised learning first then self-play outperforms emergent learning alone or fine-tuning, and introduces population-based S2P to improve performance.

ABSTRACT

A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, we investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in a simulated multi-agent environment via self-play (as done in emergent communication), and introduce the term supervised self-play (S2P) for algorithms using both of these signals. We find that first training agents via supervised learning on human data followed by self-play outperforms the converse, suggesting that it is not beneficial to emerge languages from scratch. We then empirically investigate various S2P schedules that begin with supervised learning in two environments: a Lewis signaling game with symbolic inputs, and an image-based referential game with natural language descriptions. Lastly, we introduce population based approaches to S2P, which further improves the performance over single-agent methods.

Motivation & Objective

• Investigate the data-efficiency of combining supervised language data with self-play in emergent communication.
• Determine whether starting with supervised learning before self-play improves task performance and language alignment with natural language (L*).
• Compare various S2P schedules and assess population-based approaches for robustness and performance gains.

Proposed method

• Define supervised self-play (S2P) as a combination of self-play updates and supervised updates on an expert dataset D of language L*.
• Evaluate several S2P schedules: sp2sup, sup2sp, random updates, scheduled updates (sched), and scheduled updates with speaker freezing (sched_frz).
• Study two emergent communication environments: a Lewis signaling game with symbolic inputs (Object Reconstruction) and an image-based referential game with natural language descriptions (IBR).
• Use backpropagation with straight-through Gumbel-Softmax to handle discrete speaker messages in the IBR game.
• Introduce population-based S2P (Pop-S2P) by training a population of agents and distilling into a single agent, with ensembling as an upper-bound baseline.

Experimental results

Research questions

• RQ1 Does starting with supervised learning before self-play improve sample efficiency and alignment with natural language (L*) compared to emergent learning alone?
• RQ2 How do different S2P schedules compare in terms of task performance and language drift across environments?
• RQ3 Can population-based S2P further improve performance over single-agent S2P, particularly in more complex, natural-language tasks?
• RQ4 What role does self-play play—regularization vs. constraint satisfaction—in guiding learning under S2P?
• RQ5 Are strategies like speaker freezing beneficial for mitigating language drift during self-play?

Key findings

• Supervised learning before self-play yields better performance than self-play followed by supervision in both environments.
• Adding population-based approaches (Pop-S2P) improves performance beyond single-agent S2P, especially in the image-based referential game.
• In the IBR environment, sched S2P improves over a supervised baseline, and Pop-S2P provides a further significant gain.
• Self-play acts as a form of regularization, with short-term dips in target-language performance during self-play rounds that are recovered by subsequent supervised updates.
• Emergent languages from self-play can drift away from natural language, which can be mitigated by larger initial supervised data and by population-based distillation.