[Paper Review] Entropy Minimization In Emergent Languages

Eugene Kharitonov, Rahma Chaabouni+2 more|arXiv (Cornell University)|May 31, 2019

Language and cultural evolution51 references4 citations

TL;DR

This paper investigates entropy minimization in emergent communication protocols through controlled interventions and ablation studies in the Guess Number and Image Classification tasks. It demonstrates that agents develop low-entropy, efficient messages only when necessary, with minimal reliance on messages when full input is available, and confirms robustness across architecture choices, vocabulary sizes, and sequence-based communication.

ABSTRACT

There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent's inputs and the messages is minimized, within the range afforded by the need for successful communication. That is, emergent languages are (nearly) as simple as the task they are developed for allow them to be. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.

Motivation & Objective

• To investigate the degree to which the Receiver relies on messages in emergent communication, particularly in the Guess Number task.
• To assess the robustness of entropy minimization effects under varying architectural choices, including vocabulary size and Receiver capacity.
• To evaluate whether entropy minimization persists when communication occurs via variable-length sequences of symbols, such as with GRUs or Transformers.
• To analyze the dynamics of message entropy during training to understand protocol evolution and its relationship to task success.
• To determine whether entropy minimization is a stable, general phenomenon or sensitive to hyperparameters and model capacity.

Proposed method

• Measures Receiver's dependence on messages via message shuffling: shuffled messages are fed alongside the Receiver’s own unshuffled input to compute accuracy drop.
• Uses intervention-based evaluation to isolate message reliance, comparing performance with shuffled messages versus original messages.
• Varying vocabulary size (256–4096 for Guess Number; 512–2048 for Image Classification) to test robustness of entropy minimization.
• Introduces deeper Receiver architectures (two 400×400 hidden layers) to test if model capacity affects entropy minimization.
• Employs GRU-based variable-length message generation with eos token termination and uses Stochastic Computation Graph or Gumbel-Softmax for training.
• Tracks entropy of messages $ H(m) $ during training, comparing successful and failed runs to assess entropy dynamics relative to theoretical bounds.

Experimental results

Research questions

• RQ1 How much does the Receiver rely on messages when it has access to the full input, and does message intervention significantly degrade performance?
• RQ2 Does the entropy minimization effect in emergent communication hold across different vocabulary sizes and model architectures?
• RQ3 Is entropy minimization preserved when communication occurs via variable-length sequences of symbols rather than fixed-length messages?
• RQ4 How does message entropy evolve during training, and is there a systematic relationship between entropy levels and task success?
• RQ5 Does increasing Receiver capacity alter the emergence of low-entropy, efficient communication protocols?

Key findings

• When the Receiver has access to the full input ($ m{v}_s = m{v}_r $), performance remains nearly unchanged after message shuffling, indicating minimal reliance on messages.
• Receiver accuracy remains close to the theoretical upper bound even after message shuffling, confirming that messages are not critical when input is fully available.
• Entropy minimization is robust across vocabulary sizes (256–4096 for Guess Number), with no qualitative changes in results.
• Even with a deeper Receiver architecture (two 400×400 hidden layers), message entropy remains close to the lower bound $ H_{ ext{min}} = "log_2 N_l $, indicating continued entropy minimization.
• In variable-length message settings with GRUs, entropy minimization still holds: higher entropy is used only when more input bits are hidden, and the protocol adapts accordingly.
• During training, entropy either decreases from high initial values or increases from low ones, converging near the minimum required for task success, suggesting a dynamic bottleneck emerges from coordination trade-offs.