[论文解读] Communications that Emerge through Reinforcement Learning Using a (Recurrent) Neural Network

Katsunari Shibata|arXiv (Cornell University)|Mar 10, 2017

Reinforcement Learning in Robotics参考文献 12被引用 3

一句话总结

本文展示了如何通过使用循环神经网络（RNN）的端到端强化学习，自然地涌现出多样化的通信协议——协商、信号离散化以及具身通信。智能体仅基于奖励信号学习生成和理解信号，在无需预定义符号或通信规则的情况下，实现了无碰撞导航、噪声环境下的二进制通信，以及从原始摄像头输入实现实时机器人控制。

ABSTRACT

Communication is not only an action of choosing a signal, but needs to consider the context and sensor signals. It also needs to decide what information is communicated and how it is represented in or understood from signals. Therefore, communication should be realized comprehensively together with its purpose and other functions. The recent successful results in end-to-end reinforcement learning (RL) show the importance of comprehensive learning and the usefulness of end-to-end RL. Although little is known, we have shown that a variety of communications emerge through RL using a (recurrent) neural network (NN). Here, three of them are introduced. In the 1st one, negotiation to avoid conflicts among 4 randomly-picked agents was learned. Each agent generates a binary signal from the output of its recurrent NN (RNN), and receives 4 signals from the agents three times. After learning, each agent made an appropriate final decision after negotiation for any combination of 4 agents. Differentiation of individuality among the agents also could be seen. The 2nd one focused on discretization of communication signal. A sender agent perceives the receiver's location and generates a continuous signal twice by its RNN. A receiver agent receives them sequentially, and moves according to its RNN's output to reach the sender's location. When noises were added to the signal, it was binarized through learning and 2-bit communication was established. The 3rd one focused on end-to-end comprehensive communication. A sender receives 1,785-pixel real camera image on which a real robot can be seen, and sends two sounds whose frequencies are computed by its NN. A receiver receives them, and two motion commands for the robot are generated by its NN. After learning, though some preliminary learning was necessary for the sender, the robot could reach the goal from any initial location.

研究动机与目标

• 探究在不预设符号或通信架构的前提下，综合性通信是否可通过端到端强化学习自然涌现。
• 探索智能体如何通过基于奖励的学习，自主学习协商、信号离散化以及在感知-运动任务中具身化通信。
• 证明通信协议可在仅依赖奖励信号的动态、类真实世界环境中自主学习。
• 表明RNN可在强化学习下支持结构化、有目的的通信通过自组织方式涌现。
• 建立首个基于端到端强化学习、从原始摄像头图像生成声音信号以实现目标导向导航的真实机器人系统。

提出的方法

• 智能体使用循环神经网络（RNN）根据传感器输入和内部状态生成并处理通信信号。
• 通信通过独立的、去中心化的强化学习（演员-评论家框架）学习，奖励仅在任务 episode 结束时根据任务完成情况给予。
• 在协商任务中，智能体通过三轮交互交换二进制信号，协调路径选择并避免碰撞。
• 在信号离散化任务中，训练期间引入噪声，促使RNN输出自组织为稳定的2比特二进制信号，用于传输位置信息。
• 在具身通信任务中，原始1,785像素的摄像头图像通过发送方的RNN映射为两个音调频率，接收方的RNN接收并解码这些音频信号，生成机器人运动指令。
• 通过初步学习直接机器人控制，初始化发送方网络权重，从而在真实机器人设置中实现成功通信学习。

实验结果

研究问题

• RQ1 在不预设通信协议的前提下，多个智能体间的协商能否通过端到端强化学习自然涌现？
• RQ2 在噪声条件下，连续通信信号能否通过强化学习自组织为离散且稳定的信号？
• RQ3 能否从原始感官输入（如摄像头图像）中涌现具身的、有目的的通信，并用于控制真实机器人？
• RQ4 在缺乏显式符号或架构先验的前提下，RNN在多大程度上可支持结构化、功能性通信的涌现？
• RQ5 是否仅通过奖励塑形即可学习通信协议，而无需监督或预定义信号含义？

主要发现

• 在协商任务中，智能体成功通过三轮二进制信号交互协调行动，避免碰撞，并选择非冲突路径，即使在任意智能体组合下也表现良好。
• 在训练中引入噪声后，RNN自组织产生二进制信号，形成稳定的2比特通信协议，用于传输接收方位置信息。
• 学习完成后，接收方能从任意初始位置一步抵达目标，证明该通信协议具有鲁棒性与功能性。
• 在具身通信任务中，发送方学会从原始摄像头图像生成声音信号，接收方通过实时音频输入解码并导航机器人成功抵达目标。
• 该系统仅通过端到端强化学习即实现成功机器人导航，发送方RNN学会将空间信息编码为音调频率，而无需预先定义符号映射。
• 使用直接控制学习的预训练权重显著提升了真实机器人设置中通信学习的收敛速度与成功率。