[Paper Review] Avoiding hashing and encouraging visual semantics in referential emergent language games

Daniela Mihai, Jonathon Hare|arXiv (Cornell University)|Nov 13, 2019

Language and cultural evolution13 references4 citations

TL;DR

This paper investigates how to encourage emergent communication in multi-agent referential games to capture visual semantics rather than relying on image hashing. By modifying the game with input augmentations (noise, rotation) and adding a self-supervised rotation prediction task, the authors show that end-to-end trained models can learn semantically grounded language without pretraining, achieving 84% rotation prediction accuracy and improved semantic grounding over hash-like solutions.

ABSTRACT

There has been an increasing interest in the area of emergent communication between agents which learn to play referential signalling games with realistic images. In this work, we consider the signalling game setting of Havrylov and Titov and investigate the effect of the feature extractor's weights and of the task being solved on the visual semantics learned or captured by the models. We impose various augmentation to the input images and additional tasks in the game with the aim to induce visual representations which capture conceptual properties of images. Through our set of experiments, we demonstrate that communication systems which capture visual semantics can be learned in a completely self-supervised manner by playing the right types of game.

Motivation & Objective

• Investigate whether emergent language in referential games can capture visual semantics rather than relying on image hashing.
• Examine the impact of different visual feature extractor weights (pretrained, random, learned end-to-end) on semantic grounding.
• Assess whether augmenting the game with additional tasks (e.g., rotation prediction) encourages more semantically meaningful communication.
• Determine if fully self-supervised training—without any human-annotated visual semantics—can yield semantically grounded language.
• Explore whether input-level augmentations (noise, rotation) improve the emergence of visual semantics in communication systems.

Proposed method

• Use a modified version of Havrylov and Titov’s referential game with CIFAR-10 images and a 5-token maximum message length.
• Train end-to-end models with a Straight-Through Gumbel-Softmax for discrete token generation and hinge loss to maximize correct image selection.
• Apply data augmentations (color jitter, random flip, grayscale, noise, and 90° rotations) to the sender’s input images only.
• Introduce a secondary task where one agent must predict the rotation of the sender’s input image, promoting object-level understanding.
• Train models with different CNN weight configurations: ImageNet-pretrained and fixed, randomly initialized and frozen, and end-to-end learned.
• Use BatchNorm layers after the CNN and LSTM in both sender and receiver for improved training stability and reproducibility.

Experimental results

Research questions

• RQ1 To what extent do pretrained visual feature extractors improve the visual semantics captured by emergent language in referential games?
• RQ2 Can input-level augmentations such as noise and rotation encourage agents to learn semantically meaningful representations instead of image hashing?
• RQ3 Does adding a self-supervised rotation prediction task lead to more semantically grounded communication in a multi-task learning setup?
• RQ4 Can a fully self-supervised end-to-end training procedure produce emergent language that reflects conceptual visual properties without any supervision?
• RQ5 How do different CNN weight initialization strategies (pretrained, random, learned) affect the balance between communication success and semantic grounding?

Key findings

• The pretrained and fixed VGG-16 feature extractor achieved the highest communication success rate (92%) and top-5 accuracy (99%) under noise augmentation, indicating strong semantic grounding.
• The end-to-end learned model with rotation augmentation achieved 92% communication success and 100% top-5 accuracy, showing that semantic grounding can emerge without pretraining.
• The addition of a rotation prediction task led to 84% rotation accuracy in the sender-predicts setup, suggesting the model learned to recognize object-level structure.
• Randomly initialized and frozen CNNs achieved 96% communication success with noise augmentation, indicating that strong performance can emerge without pretraining when the task is challenging.
• Communication success dropped slightly for the pretrained model under rotation augmentation (83%), suggesting that pretraining may conflict with robustness to geometric transformations.
• The end-to-end model with the rotation prediction task achieved a target-class average rank of 43.41, indicating better alignment with human notions of visual semantics than hash-based solutions.