[Paper Review] Leader-Follower Neural Networks with Local Error Signals Inspired by Complex Collectives

Chenzhong Yin, Mingxi Cheng+5 more|arXiv (Cornell University)|Oct 11, 2023

Neural Networks and Reservoir Computing4 citations

TL;DR

This paper proposes Leader-Follower Neural Networks (LFNN), a biologically inspired deep learning architecture where leaders and followers in a hierarchical structure use local error signals for training, eliminating the need for global backpropagation. LFNN-ℓ, the BP-free variant, achieves state-of-the-art performance on MNIST, CIFAR-10, and ImageNet, outperforming all prior BP-free algorithms and matching or surpassing BP-enabled baselines in key benchmarks.

ABSTRACT

The collective behavior of a network with heterogeneous, resource-limited information processing units (e.g., group of fish, flock of birds, or network of neurons) demonstrates high self-organization and complexity. These emergent properties arise from simple interaction rules where certain individuals can exhibit leadership-like behavior and influence the collective activity of the group. Motivated by the intricacy of these collectives, we propose a neural network (NN) architecture inspired by the rules observed in nature's collective ensembles. This NN structure contains workers that encompass one or more information processing units (e.g., neurons, filters, layers, or blocks of layers). Workers are either leaders or followers, and we train a leader-follower neural network (LFNN) by leveraging local error signals and optionally incorporating backpropagation (BP) and global loss. We investigate worker behavior and evaluate LFNNs through extensive experimentation. Our LFNNs trained with local error signals achieve significantly lower error rates than previous BP-free algorithms on MNIST and CIFAR-10 and even surpass BP-enabled baselines. In the case of ImageNet, our LFNN-l demonstrates superior scalability and outperforms previous BP-free algorithms by a significant margin.

Motivation & Objective

• To develop a biologically plausible deep learning architecture inspired by collective behaviors in nature, such as bird flocks and neural networks.
• To address the computational and biological implausibility of backpropagation in artificial neural networks by replacing global error signals with local learning mechanisms.
• To investigate whether local error signals in a leader-follower hierarchy can achieve competitive performance compared to backpropagation-based models.
• To evaluate the scalability and generalization of the proposed architecture across diverse datasets, including ImageNet.
• To explore the impact of leadership size on model performance and network complexity.

Proposed method

• The network is structured into hierarchical layers of leader and follower workers, where leaders have broader information access and influence the collective behavior.
• Leaders and followers use distinct local error signals derived from their respective outputs and target labels, enabling decentralized training.
• The architecture supports both BP-enabled training (LFNN) and BP-free training (LFNN-ℓ), with the latter relying solely on local loss signals.
• A leadership percentage is introduced as a hyperparameter to control the proportion of leaders in each layer, influencing information flow and learning dynamics.
• The model is trained using stochastic gradient descent with local error signals, and global loss is optionally incorporated for enhanced performance.
• Experiments are conducted on MNIST, CIFAR-10, and ImageNet using ResNet-50 and other architectures to evaluate scalability and accuracy.

Experimental results

Research questions

• RQ1 Can a leader-follower neural network architecture with local error signals achieve performance comparable to backpropagation-based models?
• RQ2 How does the proportion of leaders in the network affect model accuracy and generalization across different datasets?
• RQ3 Can a BP-free variant of the model (LFNN-ℓ) outperform existing biologically plausible, BP-free learning algorithms on standard benchmarks?
• RQ4 Does the hierarchical leader-follower structure improve scalability and performance on large-scale datasets like ImageNet?
• RQ5 What is the relationship between network depth, leadership size, and dataset complexity in terms of learning efficiency and accuracy?

Key findings

• LFNN-ℓ achieved a test error rate of 36.06% on Tiny ImageNet, outperforming all other BP-free baselines and approaching the performance of BP-enabled models.
• On the ImageNet subset, LFNN-ℓ achieved a test error rate of 53.82% with 90% leadership, surpassing all prior BP-free models and coming close to the 50.6% error rate of BP-enabled baselines.
• For MNIST and CIFAR-10, LFNN-ℓ achieved significantly lower error rates than previous BP-free algorithms, demonstrating strong generalization on small to medium-scale tasks.
• The performance of LFNN-ℓ improved with increased model parameters, indicating scalability and potential for further optimization.
• A leadership percentage of 90% yielded the best results on both Tiny ImageNet and ImageNet subset, suggesting that higher leadership is necessary for complex datasets.
• The results demonstrate that local learning with a leader-follower hierarchy can achieve competitive performance without global backpropagation, offering a biologically plausible alternative to standard training.