[Paper Review] Solving internal covariate shift in deep learning with linked neurons

Carles Roger Riera Molina, Oriol Pujol|arXiv (Cornell University)|Dec 7, 2017

Advanced Memory and Neural Computing8 references3 citations

TL;DR

This paper introduces linked neurons—a novel architecture where neurons share weights and are constrained to ensure at least one member provides non-zero gradient flow, thereby implicitly solving internal covariate shift and eliminating dying neurons. The method enables training deep networks without batch normalization or input standardization, reducing training time by up to 2× and outperforming state-of-the-art models on CIFAR-10.

ABSTRACT

This work proposes a novel solution to the problem of internal covariate shift and dying neurons using the concept of linked neurons. We define the neuron linkage in terms of two constraints: first, all neuron activations in the linkage must have the same operating point. That is to say, all of them share input weights. Secondly, a set of neurons is linked if and only if there is at least one member of the linkage that has a non-zero gradient in regard to the input of the activation function. This means that for any input in the activation function, there is at least one member of the linkage that operates in a non-flat and non-zero area. This simple change has profound implications in the network learning dynamics. In this article we explore the consequences of this proposal and show that by using this kind of units, internal covariate shift is implicitly solved. As a result of this, the use of linked neurons allows to train arbitrarily large networks without any architectural or algorithmic trick, effectively removing the need of using re-normalization schemes such as Batch Normalization, which leads to halving the required training time. It also solves the problem of the need for standarized input data. Results show that the units using the linkage not only do effectively solve the aforementioned problems, but are also a competitive alternative with respect to state-of-the-art with very promising results.

Motivation & Objective

• To address internal covariate shift and dying neurons in deep learning without relying on batch normalization or input normalization.
• To develop a neuron-level architectural mechanism that ensures consistent gradient flow during backpropagation.
• To enable training of arbitrarily deep and wide networks without architectural or algorithmic tricks.
• To evaluate the performance of linked neurons across diverse activation functions and network architectures.
• To demonstrate that linked neurons can replace batch normalization while improving training speed and generalization.

Proposed method

• Neurons are grouped into linkages where all members share the same input weights, ensuring identical operating points.
• A linkage is active if at least one neuron in the group has a non-zero gradient with respect to the input of the activation function.
• This constraint guarantees that for any input, at least one neuron operates in a non-flat, non-constant region of the activation function.
• The method implicitly stabilizes internal data distributions by maintaining non-vanishing gradients across layers.
• The approach is compatible with any activation function, including ReLU, PReLU, and SELU.
• The framework enables training without batch normalization or input standardization, reducing computational overhead.

Experimental results

Research questions

• RQ1 Can a neuron-level architecture constraint implicitly solve internal covariate shift without explicit normalization layers?
• RQ2 Does ensuring non-zero gradient flow through at least one neuron per linkage prevent dying neuron problems?
• RQ3 Can linked neurons train deep and wide networks effectively without batch normalization or input normalization?
• RQ4 How does the convergence speed of linked neurons compare to models using batch normalization?
• RQ5 Do linked neurons achieve state-of-the-art performance across diverse architectures and activation functions?

Key findings

• Linked ReLU achieved the highest accuracy on AllCNN, closely followed by PReLU, outperforming standard ReLU and SELU.
• In ResNet50, Linked SeLU outperformed all other variants, including standard SeLU and ReLU.
• Removing batch normalization from ResNet50 had minimal impact on linked neuron models, with LK-ReLU and LK-SELU maintaining high performance.
• Linked neuron models trained 2.08× faster than batch normalization when using ReLU, and 1.5× faster with SELU.
• The method enabled training with unnormalized data and without batch normalization, while maintaining or improving accuracy.
• The gradient dynamics of linked neurons were found to be more stable, allowing for faster convergence and reduced sensitivity to learning rate choices.