[Paper Review] Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity

Thomas Miconi, Aditya Rawal+2 more|arXiv (Cornell University)|Feb 24, 2020

EEG and Brain-Computer Interfaces35 citations

TL;DR

This paper introduces Backpropamine, a gradient-descent trained framework for differentiable neuromodulated plasticity, enabling self-modifying neural networks and showing gains on RL tasks and a large language-modeling task.

ABSTRACT

The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.

Motivation & Objective

• Motivate and model neuromodulated plasticity as a gradient-descent trainable mechanism inspired by biological reinforcement learning.
• Extend differentiable plasticity to include moment-to-moment neuromodulation of plasticity.
• Demonstrate performance benefits of plasticity and neuromodulation on reinforcement learning tasks and language modeling.

Proposed method

• Build on differentiable Hebbian plasticity where each connection has a fixed weight wij and a plastic component via alpha_ij Hebb_ij that updates with Hebb_ij(t+1)=Clip(Hebb_ij(t)+η x_i(t-1)x_j(t)).
• Introduce neuromodulation M(t) that scales plastic changes; simple neuromodulation replaces η with M(t) in the Hebbian update.
• Provide an alternative with eligibility traces where Hebb_ij(t+1)=Clip(Hebb_ij(t)+M(t)E_ij(t)) and E_ij(t+1)=(1-η)E_ij(t)+η x_i(t-1)x_j(t).
• Describe two task variants: simple global neuromodulation and retroactive neuromodulation via eligibility traces.
• Apply framework to reinforcement learning and language modeling tasks, including a Penn Tree Bank LSTM setup with neuromodulated plasticity.
• Note: Only recurrent weights are plastic; inputs/outputs remain non-plastic.

Experimental results

Research questions

• RQ1 Does differentiable neuromodulated plasticity improve learning performance over non-plastic and non-modulated plastic networks in RL and supervised tasks?
• RQ2 Can gradient descent optimize networks with self-modifying plasticity to achieve better language modeling perplexity on PTB?
• RQ3 How do simple neuromodulation and retroactive (eligibility-trace) modulation compare in performance and stability?
• RQ4 What are the behavioral signatures and learning dynamics produced by neuromodulated plasticity across tasks?

Key findings

• Neuromodulated plastic networks outperform non-modulated and non-plastic networks on cue-reward association tasks.
• Neuromodulation similarly improves maze navigation performance over non-modulated plasticity.
• In language modeling, LSTMs with differentiable plasticity outperform baseline LSTMs; adding neuromodulation yields additional perplexity reductions.
• Retroactive neuromodulation (eligibility traces) provides further perplexity improvements over simple neuromodulation in the large PTB model.
• Table 1 shows test perplexities where neuromodulated plastic LSTMs achieve lower values than baselines across models.