[Paper Review] Anti-efficient encoding in emergent communication

Rahma Chaabouni, Eugene Kharitonov+2 more|arXiv (Cornell University)|May 29, 2019

Language and cultural evolution38 references68 citations

TL;DR

The paper shows that two neural agents emergently develop an anti-efficient encoding where more frequent inputs map to longer messages, contrary to Zipf’s Law of Abbreviation, unless a length-penalty is added to the speaker.

ABSTRACT

Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human language. One fundamental characteristic of the latter, known as Zipf's Law of Abbreviation (ZLA), is that more frequent words are efficiently associated to shorter strings. We study whether the same pattern emerges when two neural networks, a "speaker" and a "listener", are trained to play a signaling game. Surprisingly, we find that networks develop an \emph{anti-efficient} encoding scheme, in which the most frequent inputs are associated to the longest messages, and messages in general are skewed towards the maximum length threshold. This anti-efficient code appears easier to discriminate for the listener, and, unlike in human communication, the speaker does not impose a contrasting least-effort pressure towards brevity. Indeed, when the cost function includes a penalty for longer messages, the resulting message distribution starts respecting ZLA. Our analysis stresses the importance of studying the basic features of emergent communication in a highly controlled setup, to ensure the latter will not strand too far from human language. Moreover, we present a concrete illustration of how different functional pressures can lead to successful communication codes that lack basic properties of human language, thus highlighting the role such pressures play in the latter.

Motivation & Objective

• Investigate whether emergent communication codes in neural agents follow Zipf’s Law of Abbreviation (ZLA) under power-law input distributions.
• Assess how speaker/listener pressures shape message length distributions in a simple signaling game.
• Analyze how adding a length penalty affects emergent encoding and whether it aligns with human language efficiency.
• Examine whether the emergent code resembles efficient references (optimal code) or monkey-typing distributions, and compare to natural languages.

Proposed method

• Set up a Lewis signaling game with 1000 input types drawn from a power-law distribution.
• Use a speaker and a listener, both single-layer LSTMs, to encode/decode messages ending with eos under a defined maxlen.
• Optimize via a REINFORCE-like gradient with a running baseline and entropy regularization to train the discrete message channel.
• Compare emergent encodings to reference distributions: Optimal Code (OC), Monkey Typing (MT), and natural languages across varying alphabet sizes a and maxlen.
• Experiment with and without a length penalty to test its effect on encouraging ZLA.
• Evaluate success by accuracy on all inputs after training and analyze message length distributions.

Experimental results

Research questions

• RQ1 Do emergent codes under power-law input distributions exhibit Zipf’s Law of Abbreviation or an anti-efficient pattern?
• RQ2 What biases in speaker and listener dynamics lead to the observed encoding patterns?
• RQ3 Does penalizing longer messages restore efficient encoding similar to human language (ZLA)?
• RQ4 How do emergent message properties (unigram/bigram distributions) compare to natural languages and control models?

Key findings

• Emergent communication yields anti-efficient encoding: more frequent inputs are mapped to longer messages and most messages approach maxlen.
• Untrained Speakers behave like random/uniform generators, suggesting a listener-driven effect shapes the anti-efficient pattern.
• Untrained Listeners show that emergent messages produce more discriminable representations than other encodings, indicating listener bias toward longer messages aids discrimination.
• Imposing a length-penalty on the Speaker leads to ZLA-like distributions, showing functional pressures can steer toward efficient encoding.
• Symbol-level analysis reveals emergent messages develop skewed unigram distributions and strong bigram repetition, implying phonotactic-like constraints beyond random mixtures.
• Multi-configuration experiments confirm the anti-efficient pattern is robust across settings and not typical in human language without pressures.