[Paper Review] Measuring non-trivial compositionality in emergent communication

Tomasz Korbak, Julian Zubek+1 more|arXiv (Cornell University)|Oct 28, 2020

Language and cultural evolution30 references8 citations

TL;DR

This paper evaluates seven metrics for measuring compositionality in emergent communication, demonstrating that most fail to detect non-trivial compositionality (NTC), where complex signals derive meaning through complex functions of constituents. Only tree reconstruction error (TRE), when properly parametrized, reliably detects NTC, highlighting a critical limitation in current evaluation methods for modeling language-like communication in AI systems.

ABSTRACT

Compositionality is an important explanatory target in emergent communication and language evolution. The vast majority of computational models of communication account for the emergence of only a very basic form of compositionality: trivial compositionality. A compositional protocol is trivially compositional if the meaning of a complex signal (e.g. blue circle) boils down to the intersection of meanings of its constituents (e.g. the intersection of the set of blue objects and the set of circles). A protocol is non-trivially compositional (NTC) if the meaning of a complex signal (e.g. biggest apple) is a more complex function of the meanings of their constituents. In this paper, we review several metrics of compositionality used in emergent communication and experimentally show that most of them fail to detect NTC - i.e. they treat non-trivial compositionality as a failure of compositionality. The one exception is tree reconstruction error, a metric motivated by formal accounts of compositionality. These results emphasise important limitations of emergent communication research that could hamper progress on modelling the emergence of NTC.

Motivation & Objective

• To identify and evaluate existing metrics for measuring compositionality in emergent communication models.
• To investigate why most metrics fail to detect non-trivial compositionality (NTC), where signal meaning depends on complex functions of constituents.
• To demonstrate that tree reconstruction error (TRE), when properly parametrized, is the only metric capable of reliably detecting NTC.
• To provide a unified framework for comparing compositionality metrics in emergent communication research.
• To promote methodological rigor in evaluating the emergence of complex, language-like communication in machine learning and evolutionary linguistics.

Proposed method

• The study evaluates seven compositionality metrics: context independence, topographical similarity, conflict count, tree reconstruction error (TRE), and three variants of TRE with different composition functions.
• Derivations are modeled as trees of concepts (e.g., 'blue' and 'circle'), and messages are mapped from these derivations via a learned function f: D → Θ.
• TRE measures how well a model can reconstruct the derivation tree from messages using a differentiable tree-structured model, with reconstruction error minimized via backpropagation.
• The composition function ∘ for vector representations is tested in three forms: additive, linear, and non-linear (feedforward network), to assess its impact on TRE performance.
• Experiments compare metrics across five protocols: random, holistic, trivially compositional, negation, and context-sensitive, with uniform sampling over derivations and messages.
• All metrics are evaluated under a joint uniform distribution over derivations and symbols, with scores reported across five random seeds to ensure stability.

Experimental results

Research questions

• RQ1 Do existing metrics reliably detect non-trivial compositionality (NTC), where signal meaning is a complex function of its parts?
• RQ2 Why do most compositionality metrics fail to detect NTC, and what structural flaws do they exhibit?
• RQ3 Can tree reconstruction error (TRE) detect NTC when properly parametrized, and how does its performance vary with different composition functions?
• RQ4 What are the implications of metric failure for modeling the emergence of compositional language in artificial agents?
• RQ5 How can a standardized, reliable evaluation framework for compositionality be established in emergent communication research?

Key findings

• Most compositionality metrics—context independence, topographical similarity, and conflict count—fail to detect non-trivial compositionality and instead treat it as a failure of compositionality.
• Tree reconstruction error (TRE) is the only metric that reliably detects NTC when properly parametrized, particularly with additive or linear composition functions.
• Non-linear composition in TRE leads to severe overfitting, resulting in false negatives for context-sensitive protocols, indicating that model capacity must be carefully controlled.
• The study demonstrates that additive and linear composition functions in TRE yield stable and comparable performance, while non-linear functions disrupt detection due to overfitting.
• The results reveal a critical methodological gap: many widely used metrics in emergent communication research are fundamentally flawed in detecting complex compositional structures.
• The authors provide publicly available code and reusable implementations of all metrics, enabling reproducibility and future benchmarking of compositionality in emergent communication.