[Paper Review] Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control

Guangzhi Tang, Neelesh Kumar+2 more|arXiv (Cornell University)|Oct 19, 2020

Advanced Memory and Neural Computing25 references35 citations

TL;DR

This paper introduces PopSAN, a population-coded spiking actor network trained with deep reinforcement learning, achieving comparable performance to deep actors while drastically reducing energy consumption on neuromorphic hardware (Loihi).

ABSTRACT

The energy-efficient control of mobile robots is crucial as the complexity of their real-world applications increasingly involves high-dimensional observation and action spaces, which cannot be offset by limited on-board resources. An emerging non-Von Neumann model of intelligence, where spiking neural networks (SNNs) are run on neuromorphic processors, is regarded as an energy-efficient and robust alternative to the state-of-the-art real-time robotic controllers for low dimensional control tasks. The challenge now for this new computing paradigm is to scale so that it can keep up with real-world tasks. To do so, SNNs need to overcome the inherent limitations of their training, namely the limited ability of their spiking neurons to represent information and the lack of effective learning algorithms. Here, we propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL). The population coding scheme dramatically increased the representation capacity of the network and the hybrid learning combined the training advantages of deep networks with the energy-efficient inference of spiking networks. To show the general applicability of our approach, we integrated it with a spectrum of both on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. To allow for a fair comparison among all methods, we validated them on OpenAI gym tasks. Our Loihi-run PopSAN consumed 140 times less energy per inference when compared against the deep actor network on Jetson TX2, and had the same level of performance. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.

Motivation & Objective

• Motivate energy-efficient control for mobile robots with high-dimensional observations and actions.
• Address limitations of traditional SNN training for complex continuous control tasks.
• Propose a population-coded Spiking Actor Network (PopSAN) that works with various DRL algorithms.
• Demonstrate hardware-efficient inference on Loihi while maintaining policy performance.

Proposed method

• Propose PopSAN that encodes each observation and action dimension into neuron populations with learnable receptive fields.
• Integrate PopSAN with on-policy and off-policy DRL algorithms (DDPG, TD3, SAC, PPO).
• Use gradient descent to train PopSAN parameters with DRL losses and extended spatiotemporal backpropagation for SNNs.
• Decode output population activities into continuous actions after a fixed number of timesteps.
• Deploy PopSAN on Intel Loihi with an interaction framework to enable real-time control in OpenAI Gym tasks.

Experimental results

Research questions

• RQ1 Can PopSAN match the performance of deep actor networks across multiple DRL algorithms for continuous control tasks?
• RQ2 Does population coding improve representation capacity and learning in SNNs for high-dimensional control?
• RQ3 Is PopSAN energy-efficient when deployed on neuromorphic hardware compared to conventional DNN-based controllers?
• RQ4 How does encoder learnability and population size affect performance and representation of observations and actions?

Key findings

• PopSAN achieves the same level of performance as corresponding deep actor networks across all tested DRL tasks.
• PopSAN with population coding outperforms rate-coded SNN approaches and DNN-to-SNN conversions in the experiments.
• Loihi deployment yields 140× lower energy per inference than deep actors on Jetson TX2 while maintaining performance.
• Increasing output population size improves action representation and rewards; learnable encoders outperform fixed encoders.
• PopSAN on Loihi delivers real-time control with high rewards and significantly reduced energy consumption.