[Paper Review] A New Look at Ghost Normalization

Neofytos Dimitriou, Ognjen Arandjelović|arXiv (Cornell University)|Jul 16, 2020

Advanced Neural Network Applications23 references4 citations

TL;DR

This paper introduces Sequential Normalization (SeqNorm), a novel normalization technique that sequentially applies Ghost Normalization and Group Normalization, achieving state-of-the-art performance on CIFAR-10 and CIFAR-100 without data augmentation. It identifies a unique regularization mechanism in GhostNorm and demonstrates through loss landscape visualization that GhostNorm reduces smoothness, leading to improved generalization, especially at smaller batch sizes.

ABSTRACT

Batch normalization (BatchNorm) is an effective yet poorly understood technique for neural network optimization. It is often assumed that the degradation in BatchNorm performance to smaller batch sizes stems from it having to estimate layer statistics using smaller sample sizes. However, recently, Ghost normalization (GhostNorm), a variant of BatchNorm that explicitly uses smaller sample sizes for normalization, has been shown to improve upon BatchNorm in some datasets. Our contributions are: (i) we uncover a source of regularization that is unique to GhostNorm, and not simply an extension from BatchNorm, (ii) three types of GhostNorm implementations are described, two of which employ BatchNorm as the underlying normalization technique, (iii) by visualising the loss landscape of GhostNorm, we observe that GhostNorm consistently decreases the smoothness when compared to BatchNorm, (iv) we introduce Sequential Normalization (SeqNorm), and report superior performance over state-of-the-art methodologies on both CIFAR--10 and CIFAR--100 datasets.

Motivation & Objective

• To challenge the widely held belief that BatchNorm degrades with smaller batch sizes due to poor statistical estimation.
• To uncover a unique source of regularization inherent in Ghost Normalization that is distinct from BatchNorm.
• To propose and empirically validate a new normalization technique, Sequential Normalization (SeqNorm), that outperforms existing methods on image classification benchmarks.
• To develop an efficient hyperparameter tuning strategy for SeqNorm that avoids exhaustive grid search.
• To visualize and analyze the loss landscape of GhostNorm, revealing its impact on optimization dynamics.

Proposed method

• Proposes Sequential Normalization (SeqNorm), which applies Ghost Normalization followed by Group Normalization in sequence, enabling improved feature normalization and generalization.
• Employs a two-stage hyperparameter tuning strategy: first optimizing the number of groups in GhostNorm ($G_M$), then tuning the number of groups in GroupNorm ($G_C$) using the best $G_M$.
• Utilizes gradient accumulation and multi-GPU training to simulate smaller batch sizes for GhostNorm, enabling effective training even with limited batch sizes.
• Visualizes the loss landscape of GhostNorm and observes that it consistently reduces smoothness compared to BatchNorm, especially in later training epochs.
• Introduces a novel implementation of GhostNorm that explicitly divides mini-batches into smaller groups for independent normalization, enhancing stochasticity and regularization.
• Employs a sequential normalization pipeline where the output of GhostNorm is fed into GroupNorm, allowing the network to learn more robust and disentangled feature representations.

Experimental results

Research questions

• RQ1 What causes the improved performance of Ghost Normalization despite using smaller sample sizes for statistics estimation?
• RQ2 Does Ghost Normalization introduce a unique form of regularization not present in BatchNorm?
• RQ3 Can combining Ghost Normalization and Group Normalization in sequence lead to better generalization than either method alone?
• RQ4 How does the loss landscape of GhostNorm compare to that of BatchNorm, and what does this imply for optimization dynamics?
• RQ5 Can a more efficient hyperparameter tuning strategy be developed for SeqNorm that avoids full grid search while maintaining performance?

Key findings

• Ghost Normalization achieves a test accuracy of 82.8% on CIFAR-100 with $G_M = 8$, outperforming BatchNorm (82.1%) despite using only 4 samples per group for normalization.
• Sequential Normalization (SeqNorm) with $G_C = 4$ and $G_M = 8$ achieves a test accuracy of 83.8% on CIFAR-100, surpassing the current SOTA on this dataset without data augmentation.
• On CIFAR-10, SeqNorm with $G_C = 16$ and $G_M = 8$ achieves a test accuracy of 97.4%, matching the current SOTA while using a simpler training setup.
• Loss landscape visualization shows that GhostNorm consistently reduces smoothness compared to BatchNorm, especially in later training epochs, indicating a more complex but potentially more generalizable optimization path.
• The proposed sequential tuning strategy for $G_M$ and $G_C$ reduces time complexity to $\Theta(G_C + G_M)$, enabling faster hyperparameter search than full grid search.
• Experiments show that reversing the order of GhostNorm and GroupNorm in SeqNorm, or combining them simultaneously, leads to worse performance, confirming the importance of sequential application.