[Paper Review] Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware

Guangzhi Tang, Neelesh Kumar+1 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Memory and Neural Computing27 references13 citations

TL;DR

This paper proposes Spiking DDPG (SDDPG), a hybrid reinforcement learning framework that jointly trains a spiking actor network (SAN) and a deep critic network for energy-efficient, mapless navigation. Deployed on Intel's Loihi neuromorphic chip, SDDPG achieves 75× lower energy consumption per inference than DDPG on Jetson TX2 while improving navigation success rates by 1%–4.2%, demonstrating superior energy efficiency and performance through synergistic co-learning between SNN and DNN components.

ABSTRACT

Energy-efficient mapless navigation is crucial for mobile robots as they explore unknown environments with limited on-board resources. Although the recent deep reinforcement learning (DRL) approaches have been successfully applied to navigation, their high energy consumption limits their use in several robotic applications. Here, we propose a neuromorphic approach that combines the energy-efficiency of spiking neural networks with the optimality of DRL and benchmark it in learning control policies for mapless navigation. Our hybrid framework, spiking deep deterministic policy gradient (SDDPG), consists of a spiking actor network (SAN) and a deep critic network, where the two networks were trained jointly using gradient descent. The co-learning enabled synergistic information exchange between the two networks, allowing them to overcome each other's limitations through a shared representation learning. To evaluate our approach, we deployed the trained SAN on Intel's Loihi neuromorphic processor. When validated on simulated and real-world complex environments, our method on Loihi consumed 75 times less energy per inference as compared to DDPG on Jetson TX2, and also exhibited a higher rate of successful navigation to the goal, which ranged from 1% to 4.2% and depended on the forward-propagation timestep size. These results reinforce our ongoing efforts to design brain-inspired algorithms for controlling autonomous robots with neuromorphic hardware.

Motivation & Objective

• To address the high energy consumption of deep reinforcement learning (DRL) in mobile robot navigation under limited on-board resources.
• To overcome the limitations of purely spiking neural networks (SNNs), such as poor representation of high-precision action-values and susceptibility to catastrophic forgetting in complex environments.
• To develop a hybrid framework that leverages the energy efficiency of SNNs and the policy optimization strength of DRL for real-time, mapless navigation.
• To enable efficient deployment of reinforcement learning policies on neuromorphic hardware like Intel's Loihi by training SNNs directly with gradient descent.
• To achieve both higher navigation success rates and significantly lower energy consumption than state-of-the-art DRL methods on edge devices.

Proposed method

• Proposes a hybrid SNN/DNN framework, Spiking DDPG (SDDPG), with a spiking actor network (SAN) for action selection and a deep critic network for action-value evaluation.
• Employs a modified spatiotemporal backpropagation (STBP) algorithm to enable end-to-end gradient-based training of the spiking actor network.
• Uses joint training of the SAN and deep critic network via gradient descent, enabling shared representation learning and mutual improvement.
• Deploys the trained SAN on Intel's Loihi neuromorphic processor, exploiting its event-based, asynchronous computation for low-power inference.
• Optimizes inference energy by training the SNN for low time-step values (T=5 to 50), minimizing latency and power without sacrificing performance.
• Employs a reward-modulated learning rule with memory replay to mitigate catastrophic forgetting, combining biological plausibility with DRL robustness.

Experimental results

Research questions

• RQ1 Can a jointly trained hybrid SNN/DNN framework outperform standalone DRL or SNN methods in energy efficiency and navigation accuracy for mapless robot control?
• RQ2 To what extent can spiking neural networks trained via gradient descent achieve high-precision action-value representation comparable to deep neural networks?
• RQ3 How does the inference energy consumption of a co-trained SNN on neuromorphic hardware compare to that of a DNN on conventional edge devices in real-world navigation tasks?
• RQ4 Does the inherent noise in SNNs’ spatiotemporal representations help escape poor local minima during training, improving policy optimization?
• RQ5 Can direct training of SNNs with STBP enable lower time-step inference (T) while maintaining performance, unlike DNN-to-SNN conversion methods?

Key findings

• SDDPG deployed on Intel's Loihi neuromorphic processor consumed 75 times less energy per inference than DDPG running on Jetson TX2's energy-efficient mode (MAXQ).
• The SDDPG method achieved a 1% to 4.2% higher success rate in navigating to the goal compared to DDPG, depending on the forward-propagation timestep size (T).
• SDDPG with T=5 achieved 15.53 mJ per inference on Loihi, while the equivalent performance using DNN-to-SNN conversion required 4.5× more energy and 5× more timesteps.
• The hybrid co-learning approach enabled the SNN to overcome its limitation in representing high-precision action-values, leading to better policy optimization than standalone SNNs.
• The SNN's noisy spatiotemporal input representation contributed to escaping suboptimal local minima, enhancing training robustness.
• SDDPG demonstrated superior performance and energy efficiency compared to state-of-the-art methods, marking one of the first cases where an energy-efficient neuromorphic method outperforms the current SOTA in accuracy for robotic navigation tasks.