[Paper Review] Backpropagation and Biological Plausibility

Alessandro Betti, Marco Gori+1 more|arXiv (Cornell University)|Aug 21, 2018

Neural Networks and Applications2 references3 citations

TL;DR

This paper proposes a biologically plausible alternative to backpropagation by formulating supervised learning as a saddle-point search in a Lagrangian adjoint space of weights, activations, and multipliers. By updating weights via gradient descent, activations via gradient descent, and multipliers via gradient ascent, the method achieves fully local learning dynamics that avoid the non-local weight transport problem of backpropagation, while enabling automatic architecture construction through support neuron identification via ℓ₁ regularization.

ABSTRACT

By and large, Backpropagation (BP) is regarded as one of the most important neural computation algorithms at the basis of the progress in machine learning, including the recent advances in deep learning. However, its computational structure has been the source of many debates on its arguable biological plausibility. In this paper, it is shown that when framing supervised learning in the Lagrangian framework, while one can see a natural emergence of Backpropagation, biologically plausible local algorithms can also be devised that are based on the search for saddle points in the learning adjoint space composed of weights, neural outputs, and Lagrangian multipliers. This might open the doors to a truly novel class of learning algorithms where, because of the introduction of the notion of support neurons, the optimization scheme also plays a fundamental role in the construction of the architecture.

Motivation & Objective

• To address the long-standing criticism that backpropagation is biologically implausible due to its reliance on non-local weight transport.
• To develop a learning algorithm that maintains the optimization power of backpropagation while ensuring all updates are local and computationally feasible in a biological context.
• To explore how the Lagrangian framework can naturally lead to the emergence of support neurons and sparse network architectures through ℓ₁ regularization.
• To demonstrate that the search for saddle points in the adjoint space (weights, activations, Lagrange multipliers) yields a fully local learning algorithm with the same asymptotic complexity as backpropagation.

Proposed method

• Formulate supervised learning as a constrained optimization problem using a Lagrangian that enforces neural network dynamics as equality constraints.
• Define the Lagrangian with terms for loss, ℓ₁ regularization on hidden neuron activations, and Lagrange multipliers enforcing the neural computation constraints.
• Derive the stationarity conditions of the Lagrangian to obtain gradient updates: descent on weights and activations, ascent on multipliers.
• Implement a joint update rule where each neuron locally computes its own gradient with respect to its inputs and outputs, avoiding global backpropagation.
• Introduce ℓ₁ regularization on hidden neuron activations to distinguish between support neurons (λᵢᵏ ≠ 0) and straw neurons (λᵢᵏ = 0), enabling automatic pruning.
• Show that the algorithm operates in Θ(|𝒜|·ℓ) time, matching the asymptotic complexity of standard backpropagation, while being fully local.

Experimental results

Research questions

• RQ1 Can a learning algorithm based on the Lagrangian formalism achieve the same optimization performance as backpropagation while being fully local?
• RQ2 How can the notion of support neurons emerge naturally from the Lagrangian saddle-point framework, and what role do they play in network architecture construction?
• RQ3 Does the proposed method eliminate the need for non-local weight transport, thereby resolving a key objection to the biological plausibility of backpropagation?
• RQ4 Can the algorithm be extended to online and mini-batch learning while preserving its local update structure?
• RQ5 How does ℓ₁ regularization on hidden activations lead to the emergence of sparse, pruned architectures through the identification of support neurons?

Key findings

• The proposed algorithm achieves fully local learning dynamics by updating weights via gradient descent, activations via gradient descent, and Lagrange multipliers via gradient ascent, eliminating the need for non-local weight transport.
• The stationarity conditions of the Lagrangian recover the classic backpropagation factorization when αₖⱼ = 0, showing consistency with standard backpropagation in the absence of regularization.
• With ℓ₁ regularization, the algorithm naturally identifies support neurons (where λₖᵢ ≠ 0) and straw neurons (where λₖᵢ = 0), enabling automatic network pruning and architecture construction.
• The algorithm maintains the same asymptotic time complexity of Θ(|𝒜|·ℓ) as standard backpropagation, ensuring computational efficiency.
• The method is extendable to recurrent networks and structured data (e.g., graphs) by expressing data-induced constraints within the Lagrangian framework.
• The algorithm's local nature suggests it may be biologically plausible, particularly in batch mode or in tasks with inherent temporal structure, though online extension requires prediction of intermediate variables.