[Paper Review] BCN: Batch Channel Normalization for Image Classification

Afifa Khaled, Chao Li+2 more|arXiv (Cornell University)|Dec 1, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Batch Channel Normalization (BCN), a novel normalization technique that jointly exploits batch and channel dimensions by separately normalizing along (N, H, W) and (C, H, W) axes, then adaptively fusing the results using learnable parameters. BCN achieves superior generalization and training speed across CNNs and Vision Transformers, outperforming BN, LN, and other baselines on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

ABSTRACT

Normalization techniques have been widely used in the field of deep learning due to their capability of enabling higher learning rates and are less careful in initialization. However, the effectiveness of popular normalization technologies is typically limited to specific areas. Unlike the standard Batch Normalization (BN) and Layer Normalization (LN), where BN computes the mean and variance along the (N,H,W) dimensions and LN computes the mean and variance along the (C,H,W) dimensions (N, C, H and W are the batch, channel, spatial height and width dimension, respectively), this paper presents a novel normalization technique called Batch Channel Normalization (BCN). To exploit both the channel and batch dependence and adaptively and combine the advantages of BN and LN based on specific datasets or tasks, BCN separately normalizes inputs along the (N, H, W) and (C, H, W) axes, then combines the normalized outputs based on adaptive parameters. As a basic block, BCN can be easily integrated into existing models for various applications in the field of computer vision. Empirical results show that the proposed technique can be seamlessly applied to various versions of CNN or Vision Transformer architecture. The code is publicly available at https://github.com/AfifaKhaled/BatchChannel-Normalization

Motivation & Objective

• To address the limitations of Batch Normalization (BN) in small-batch settings and Layer Normalization (LN) in convolutional layers.
• To develop a normalization technique that leverages both batch and channel dependencies for improved feature representation.
• To create a general-purpose normalization layer that enhances training stability, convergence speed, and model generalization across diverse architectures.
• To enable effective use of normalization in online learning and distributed training scenarios where BN is restricted.
• To empirically validate BCN's superiority over BN, LN, IN, and GN across multiple vision benchmarks and model architectures.

Proposed method

• BCN computes mean and variance along the (N, H, W) axis (similar to BN) and along the (C, H, W) axis (similar to LN) separately.
• It applies batch normalization-style normalization on the (N, H, W) dimension and layer normalization-style on the (C, H, W) dimension.
• The two normalized outputs are combined using adaptive, learnable parameters that dynamically weight the contributions of each normalized stream.
• The adaptive fusion mechanism allows BCN to automatically prioritize the most relevant statistical statistics based on input data distribution.
• BCN is plug-and-play, requiring minimal architectural changes and compatible with existing CNN and Vision Transformer models.
• The method uses a small epsilon for numerical stability and includes learnable scale (γ) and shift (β) parameters for feature reparameterization.

Figure 1 : Visualization on several normalization techniques. Each subplot shows a feature map tensor with N the batch axes, C the channel axes, and (H, W) the spatial height and width axes.

Experimental results

Research questions

• RQ1 Can a normalization technique that jointly models batch and channel dependencies outperform standard BN and LN in image classification?
• RQ2 Does BCN maintain strong performance under small batch sizes, where BN typically degrades?
• RQ3 Can BCN improve training speed and generalization in both convolutional networks and Vision Transformers?
• RQ4 How does adaptive fusion of batch and channel statistics affect model convergence and accuracy across different datasets?
• RQ5 Is BCN effective in self-supervised learning and large-scale vision tasks like ImageNet?

Key findings

• On CIFAR-10, BCN achieved 84.16% validation accuracy after 20 epochs, outperforming BN (78.58%) and LN (79.74%) under the same settings.
• On CIFAR-100, BCN achieved state-of-the-art performance, significantly improving generalization over BN, LN, IN, and GN.
• In Vision Transformer (ViT) experiments, replacing BN with BCN improved test accuracy by 0.72% and training accuracy by 0.73% on CIFAR-100.
• BCN maintained strong performance even at very small batch sizes (e.g., 2 and 4), where BN typically fails, achieving 81.42% validation accuracy at batch size 2.
• On ImageNet with DenseNet-201, BCN demonstrated superior training and test accuracy compared to BN, as shown in Figures 10 and 11.
• Ablation studies confirmed that direct normalization over (N, C, H, W) is less effective than the two-stream adaptive fusion approach used in BCN.

Figure 10 : Training accuracy of DenseNet with ImageNet dataset.