[Paper Review] Communications that Emerge through Reinforcement Learning Using a (Recurrent) Neural Network

Katsunari Shibata|arXiv (Cornell University)|Mar 10, 2017

Reinforcement Learning in Robotics12 references3 citations

TL;DR

This paper demonstrates how diverse communication protocols—negotiation, signal discretization, and grounded communication—emerge organically through end-to-end reinforcement learning using recurrent neural networks (RNNs). Agents learn to generate and interpret signals based solely on reward signals, achieving collision-free navigation, binary communication under noise, and real-time robot control from raw camera input, all without predefined symbols or communication rules.

ABSTRACT

Communication is not only an action of choosing a signal, but needs to consider the context and sensor signals. It also needs to decide what information is communicated and how it is represented in or understood from signals. Therefore, communication should be realized comprehensively together with its purpose and other functions. The recent successful results in end-to-end reinforcement learning (RL) show the importance of comprehensive learning and the usefulness of end-to-end RL. Although little is known, we have shown that a variety of communications emerge through RL using a (recurrent) neural network (NN). Here, three of them are introduced. In the 1st one, negotiation to avoid conflicts among 4 randomly-picked agents was learned. Each agent generates a binary signal from the output of its recurrent NN (RNN), and receives 4 signals from the agents three times. After learning, each agent made an appropriate final decision after negotiation for any combination of 4 agents. Differentiation of individuality among the agents also could be seen. The 2nd one focused on discretization of communication signal. A sender agent perceives the receiver's location and generates a continuous signal twice by its RNN. A receiver agent receives them sequentially, and moves according to its RNN's output to reach the sender's location. When noises were added to the signal, it was binarized through learning and 2-bit communication was established. The 3rd one focused on end-to-end comprehensive communication. A sender receives 1,785-pixel real camera image on which a real robot can be seen, and sends two sounds whose frequencies are computed by its NN. A receiver receives them, and two motion commands for the robot are generated by its NN. After learning, though some preliminary learning was necessary for the sender, the robot could reach the goal from any initial location.

Motivation & Objective

• To investigate whether comprehensive communication can emerge through end-to-end reinforcement learning without predefined symbols or communication architectures.
• To explore how agents can learn to negotiate, discretize signals, and ground communication in sensorimotor tasks through reward-based learning.
• To demonstrate that communication protocols can be learned autonomously in dynamic, real-world-like environments using only reward signals.
• To show that RNNs can support the emergence of structured, purposeful communication through self-organization under RL.
• To establish the first real-world robotic system using end-to-end RL to generate sound signals from raw camera images for goal-directed navigation.

Proposed method

• Agents use recurrent neural networks (RNNs) to generate and process communication signals based on sensor inputs and internal states.
• Communication is learned via independent, decentralized reinforcement learning (actor-critic framework), with rewards given only at the end of an episode based on task success.
• In negotiation tasks, agents exchange binary signals over three rounds to coordinate path choices and avoid collisions.
• For signal discretization, noise is introduced during training, causing RNN outputs to self-organize into stable 2-bit binary signals for location transmission.
• In grounded communication, raw 1,785-pixel camera images are mapped to two sound frequencies via the sender’s RNN, which are received and decoded by the receiver’s RNN to generate robot motion commands.
• Preliminary learning of direct robot control is used to initialize the sender’s network weights, enabling successful communication learning in the real robot setup.

Experimental results

Research questions

• RQ1 Can negotiation among multiple agents emerge through end-to-end reinforcement learning without predefined communication protocols?
• RQ2 Can continuous communication signals self-organize into discrete, robust signals under noisy conditions through RL?
• RQ3 Can grounded, purposeful communication emerge from raw sensory input (e.g., camera images) and be used to control a real robot?
• RQ4 To what extent can RNNs support the emergence of structured, functional communication without explicit symbolic or architectural priors?
• RQ5 Can communication protocols be learned through reward shaping alone, without supervision or predefined signal meanings?

Key findings

• In the negotiation task, agents successfully coordinated to avoid collisions and select non-conflicting paths through 3 rounds of binary signaling, even with arbitrary agent combinations.
• When noise was introduced during training, the RNNs self-organized to produce binary signals, resulting in a stable 2-bit communication protocol for transmitting receiver location.
• After learning, the receiver could reach the goal in one step from any initial position, demonstrating that the communication protocol was both robust and functional.
• In the grounded communication task, the sender learned to generate sound signals from raw camera images, and the receiver interpreted them to navigate the robot to the goal using real-time audio input.
• The system achieved successful robot navigation using only end-to-end RL, with the sender’s RNN learning to encode spatial information into sound frequencies without prior symbolic mapping.
• The use of pre-trained weights from direct control learning significantly improved the convergence and success rate of communication learning in the real-world robot setup.