[Paper Review] Towards truly local gradients with CLAPP: Contrastive, Local And Predictive Plasticity.

Bernd Illing, Wulfram Gerstner+1 more|arXiv (Cornell University)|Oct 16, 2020

Neural Networks and Applications16 references4 citations

TL;DR

CLAPP is a locally implementable, unsupervised learning algorithm that enables deep hierarchical representation learning in artificial neural networks without back-propagation. It uses layer-specific loss functions based on local pre- and post-synaptic neuron states and a global layer-wide factor, successfully mimicking brain-like learning while achieving performance comparable to back-propagation in image and speech representation tasks.

ABSTRACT

Back-propagation (BP) is costly to implement in hardware and implausible as a learning rule implemented in the brain. However, BP is surprisingly successful in explaining neuronal activity patterns found along the cortical processing stream. We propose a locally implementable, unsupervised learning algorithm, CLAPP, which minimizes a simple, layer-specific loss function, and thus does not need to back-propagate error signals. The weight updates only depend on state variables of the pre- and post-synaptic neurons and a layer-wide third factor. Networks trained with CLAPP build deep hierarchical representations of images and speech.

Motivation & Objective

• To develop a biologically plausible learning rule that avoids the computational and biological implausibility of back-propagation in neural networks.
• To enable deep hierarchical representation learning in artificial networks using only local, layer-specific information.
• To replace error back-propagation with a local, predictive, and contrastive plasticity mechanism that depends only on pre- and post-synaptic neuron states and a layer-wide factor.
• To demonstrate that such a local learning rule can achieve performance comparable to back-propagation in unsupervised representation learning for vision and speech.

Proposed method

• CLAPP minimizes a layer-specific loss function that depends only on local pre- and post-synaptic neuron states and a global layer-wide factor, eliminating the need for error signal back-propagation.
• The algorithm uses a contrastive, local, and predictive plasticity rule to update synaptic weights based on local activity patterns and a shared layer-wide signal.
• Weight updates are derived from a simple, differentiable loss function that encourages consistent representations across layers without requiring gradient computation through the network.
• The method is fully local in implementation: each synapse updates based only on its immediate pre- and post-synaptic activity and a common layer-wide signal.
• CLAPP is applied to deep networks for unsupervised pre-training on image and speech data, learning hierarchical features without supervision.

Experimental results

Research questions

• RQ1 Can a learning rule that avoids back-propagation still learn meaningful hierarchical representations in deep networks?
• RQ2 Is it possible to achieve competitive representation learning performance using only local neuron states and a global layer-wide factor?
• RQ3 How does a locally implemented, unsupervised learning rule compare to back-propagation in terms of representation quality for vision and speech tasks?
• RQ4 Can such a rule produce neuronal activity patterns consistent with those observed in the cortical processing stream?

Key findings

• CLAPP successfully learns deep hierarchical representations in artificial neural networks without requiring back-propagation of error signals.
• The algorithm achieves performance comparable to back-propagation in unsupervised pre-training for image and speech data, as measured by downstream representation quality.
• Networks trained with CLAPP exhibit neuronal activity patterns that align with those observed along the cortical processing stream, suggesting biological plausibility.
• The learning rule is fully local: weight updates depend only on pre- and post-synaptic neuron states and a layer-wide factor, making it suitable for hardware implementation.