[论文解读] Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control

Guangzhi Tang, Neelesh Kumar+2 more|arXiv (Cornell University)|Oct 19, 2020

Advanced Memory and Neural Computing参考文献 25被引用 35

一句话总结

本论文提出 PopSAN，一种用深度强化学习训练的人口编码脉冲神经元网络，在实现与深度 Actor 相当的性能的同时，在神经形态硬件 Loihi 上显著降低能耗。

ABSTRACT

The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-world tasks. To do so, SNNs need to overcome the inherent limitations of their training, namely the limited ability of their spiking neurons to represent information and the lack of effective learning algorithms. Here, we propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL). The population coding scheme dramatically increased the representation capacity of the network and the hybrid learning combined the training advantages of deep networks with the energy-efficient inference of spiking networks. To show the general applicability of our approach, we integrated it with a spectrum of both on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. To allow for a fair comparison among all methods, we validated them on OpenAI gym tasks. Our Loihi-run PopSAN consumed 140 times less energy per inference when compared against the deep actor network on Jetson TX2, and had the same level of performance. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.

研究动机与目标

• 在具有高维观测和动作的移动机器人上，推动能效高的控制。
• 解决传统 SNN 训练在复杂连续控制任务中的局限性。
• 提出一个与多种 DRL 算法兼容的群体编码脉冲神经元演员网络（PopSAN）。
• 在 Loihi 上演示硬件高效推理，同时维持策略性能。

提出的方法

• 提出 PopSAN，将每个观测和动作维度编码为具有可学习感受野的神经元群体。
• 将 PopSAN 与基于策略和离策略的 DRL 算法（DDPG、TD3、SAC、PPO）集成。
• 使用梯度下降来训练 PopSAN 的参数，结合 DRL 损失和用于SNN的扩展时空反向传播。
• 在固定的时间步数后，将输出群体活动解码为连续动作。
• 在 Intel Loihi 上部署 PopSAN，并使用交互框架实现 OpenAI Gym 任务的实时控制。

实验结果

研究问题

• RQ1 PopSAN 是否能在多种 DRL 算法下达到与深度 Actor 网络在连续控制任务中的性能相当？
• RQ2 群体编码是否提升高维控制下 SNN 的表征能力与学习效果？
• RQ3 将 PopSAN 部署到神经形态硬件上相比于传统的基于 DNN 的控制器，是否能源高效？
• RQ4 编码器的可学习性和群体规模如何影响观测与动作的表现与表征？

主要发现

• PopSAN 在所有测试的 DRL 任务中达到与相应的深度 Actor 网络相同水平的性能。
• 在实验中，采用群体编码的 PopSAN 超越了基于速率编码的 SNN 方法以及 DNN 转换为 SNN 的方法。
• 在 Loihi 上部署的能耗在推理单次相比 Jetson TX2 的深度演员下降 140 倍，同时保持性能。
• 增大输出群体规模可以改善动作表示和奖励；可学习编码器优于固定编码器。
• 在 Loihi 上的 PopSAN 实现了高奖励的实时控制，并显著降低能耗。