[Paper Review] Feedback alignment in deep convolutional networks

Theodore Moskovitz, Ashok Litwin-Kumar+1 more|arXiv (Cornell University)|Dec 12, 2018

Advanced Memory and Neural Computing25 references45 citations

TL;DR

The paper extends feedback alignment (FA) to deep convolutional nets, introducing sign-concordant feedback and normalization strategies that enable FA to achieve competitive ImageNet performance compared with backpropagation on various CNN architectures.

ABSTRACT

Ongoing studies have identified similarities between neural representations in biological networks and in deep artificial neural networks. This has led to renewed interest in developing analogies between the backpropagation learning algorithm used to train artificial networks and the synaptic plasticity rules operative in the brain. These efforts are challenged by biologically implausible features of backpropagation, one of which is a reliance on symmetric forward and backward synaptic weights. A number of methods have been proposed that do not rely on weight symmetry but, thus far, these have failed to scale to deep convolutional networks and complex data. We identify principal obstacles to the scalability of such algorithms and introduce several techniques to mitigate them. We demonstrate that a modification of the feedback alignment method that enforces a weaker form of weight symmetry, one that requires agreement of weight sign but not magnitude, can achieve performance competitive with backpropagation. Our results complement those of Bartunov et al. (2018) and Xiao et al. (2018b) and suggest that mechanisms that promote alignment of feedforward and feedback weights are critical for learning in deep networks.

Motivation & Objective

• Motivate biologically plausible credit assignment by relaxing weight symmetry in learning rules.
• Investigate why original FA struggles with deep CNNs and complex data.
• Develop modifications to FA to improve performance and stability in deep architectures.
• Examine the impact of fixed excitatory/inhibitory connectivity and weight-norm constraints on learning.

Proposed method

• Apply feedback alignment in convolutional layers using a separate backward feedback matrix B for error propagation.
• Introduce sign-concordant feedback (uSF) where B is aligned with the sign of forward weights W.
• Propose normalization strategies to control gradient magnitudes (SN, Init.) and maintain alignment.
• Explore fixed excitatory/inhibitory (E/I) connectivity and its effect on learning.
• Evaluate FA variants on MNIST, CIFAR-10, and ImageNet across multiple architectures.
• Compare FA variants to BP, DFA, and DenseFA; include constant-norm experiments to isolate effects.

Experimental results

Research questions

• RQ1 Can FA-scale methods match backpropagation performance in deep CNNs on standard vision benchmarks?
• RQ2 Do sign-concordant feedback and gradient normalization improve FA’s scalability to deep architectures?
• RQ3 What is the impact of constraining weight norms or fixing excitatory/inhibitory signs on learning with FA?
• RQ4 Are there biological plausibility scenarios (e.g., E/I constraints) that still support effective learning with FA?

Key findings

• FA variants with sign-concordant feedback and normalization achieve competitive top-1 errors versus BP on several datasets.
• On ImageNet, FA with sign-concordant feedback and initialization/norm control narrows the gap to BP, achieving close performance under certain configurations.
• Different FA adaptations (Init., SN, E/I, Const.) lead to varying gains; some methods substantially reduce the FA–BP gap (e.g., FA-uSF Init. and SN).
• Constraining forward weight norms can improve FA performance, with FA-uSF Const achieving Top-1 of 51.2% on ImageNet in the reported setup.
• Fixed weight sign constraints (E/I) markedly impede learning, demonstrating the importance of flexible sign dynamics for FA performance.
• Direct comparisons show DFA and DenseFA can deliver competitive results in limited contexts, but memory constraints restrict their use on large models.