[Paper Review] Long short-term memory and learning-to-learn in networks of spiking neurons

Guillaume Bellec, Darjan Salaj+3 more|arXiv (Cornell University)|Mar 26, 2018

Advanced Memory and Neural ComputingEngineering164 citations

TL;DR

LSNNs with adapting neurons trained via BPTT and DEEP R achieve LSTM-like performance on sequential MNIST and TIMIT; with Learning-to-Learn (L2L) they acquire fast-learning priors and enable meta-reinforcement learning.

ABSTRACT

Recurrent networks of spiking neurons (RSNNs) underlie the astounding computing and learning capabilities of the brain. But computing and learning capabilities of RSNN models have remained poor, at least in comparison with artificial neural networks (ANNs). We address two possible reasons for that. One is that RSNNs in the brain are not randomly connected or designed according to simple rules, and they do not start learning as a tabula rasa network. Rather, RSNNs in the brain were optimized for their tasks through evolution, development, and prior experience. Details of these optimization processes are largely unknown. But their functional contribution can be approximated through powerful optimization methods, such as backpropagation through time (BPTT). A second major mismatch between RSNNs in the brain and models is that the latter only show a small fraction of the dynamics of neurons and synapses in the brain. We include neurons in our RSNN model that reproduce one prominent dynamical process of biological neurons that takes place at the behaviourally relevant time scale of seconds: neuronal adaptation. We denote these networks as LSNNs because of their Long short-term memory. The inclusion of adapting neurons drastically increases the computing and learning capability of RSNNs if they are trained and configured by deep learning (BPTT combined with a rewiring algorithm that optimizes the network architecture). In fact, the computational performance of these RSNNs approaches for the first time that of LSTM networks. In addition RSNNs with adapting neurons can acquire abstract knowledge from prior learning in a Learning-to-Learn (L2L) scheme, and transfer that knowledge in order to learn new but related tasks from very few examples. We demonstrate this for supervised learning and reinforcement learning.

Motivation & Objective

• Motivate RSNN limitations compared to ANNs and explore optimization by evolution, development, and prior experience.
• Introduce LSNNs by adding neuronal adaptation to RSNNs to extend short-term memory.
• Show that BPTT combined with DEEP R can train LSNNs for complex tasks.
• Demonstrate learning-to-learn (L2L) enabling rapid adaptation to new tasks.
• Demonstrate meta-reinforcement learning (meta-RL) capabilities in LSNNs and potential neuromorphic implications.

Proposed method

• Introduce LSNN architecture with two neuron populations: regular LIF and adapting LIF neurons.
• Train LSNNs with backpropagation through time (BPTT) using a dampened pseudo-derivative for spikes.
• Combine BPTT with DEEP R for synaptic rewiring to optimize connectivity.
• Evaluate on sequential MNIST and TIMIT to compare with LSTM and RNN baselines.
• Apply Learning-to-Learn (L2L) with an outer loop optimizing hyper-parameters to support fast inner-loop learning.
• Demonstrate meta-reinforcement learning by training LSNNs to perform reward-based navigation tasks.

Experimental results

Research questions

• RQ1 Can LSNNs close the performance gap to LSTM networks on time-series classification tasks like sequential MNIST and TIMIT?
• RQ2 Does incorporating neuronal adaptation extend short-term memory in RSNNs effectively for complex tasks?
• RQ3 Can learning-to-learn (L2L) imprint priors in LSNNs to enable rapid learning of new tasks from few examples?
• RQ4 Can LSNNs exhibit meta-reinforcement learning capabilities with reward signals without changing synaptic weights?

Key findings

• LSNNs achieve 94.7% and 96.4% accuracy on sequential MNIST for 1 ms and 2 ms pixel presentation, respectively, approaching LSTM performance (98.5% and 98.0%).
• On TIMIT, LSNNs reach 33.2% classification error, below the mean around 40% from 200 LSTM trials but above the best 20 LSTM trials (29.7%).
• DEEP R–enabled sparse LSNNs (about 12% connectivity) can outperform fully connected LSNNs and approach LSTM metrics.
• Learning-to-Learn (L2L) enables LSNNs to learn new nonlinear functions from a teacher with few trials, surpassing linear predictors and showing rapid internal model formation.
• LSNNs trained with meta-RL acquire abstract knowledge for navigation and planning, demonstrating reward-based learning abilities in sparse RSNNs.
• L2L installs priors for smooth function classes (e.g., nonlinear TNs, sinus functions) into LSNNs, enabling efficient online learning without synaptic weight changes.