[Paper Review] High-Performance Large-Scale Image Recognition Without Normalization

Andrew Brock, Soham De+2 more|arXiv (Cornell University)|Feb 11, 2021

Advanced Neural Network Applications84 references256 citations

TL;DR

The paper introduces Adaptive Gradient Clipping (AGC) and Normalizer-Free Network (NFNet) architectures that achieve state-of-the-art ImageNet results without normalization layers, matching or surpassing EfficientNets while training much faster, and benefiting from strong data augmentation and SAM finish for largest models.

ABSTRACT

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

Motivation & Objective

• Understand limitations of batch normalization in large-scale image classification.
• Develop a normalization-free training regime that maintains competitive accuracy.
• Enable stable large-batch training with minimal hyperparameter tuning.
• Design NFNet architectures optimized for training speed on modern accelerators.
• Demonstrate transfer learning and fine-tuning advantages after large-scale pre-training.

Proposed method

• Propose Adaptive Gradient Clipping (AGC) that clips unit-wise gradient norms relative to corresponding weight norms to stabilize training.
• Adopt and refine Normalizer-Free ResNet (NFNet) blocks with variance-preserving initialization and Scaled Weight Standardization.
• Incorporate learnable skip-init scalars and adjusted squeeze-excite scaling to control activation statistics.
• Design a family of NFNet architectures (F0–F6) with depth-, width-, and resolution-scaling strategies focused on training latency.
• Apply strong regularization and data augmentation (MixUp, RandAugment, CutMix) to counter overfitting in non-normalized nets.
• Evaluate performance on ImageNet with large batch sizes (1024–4096) and compare against batch-normalized counterparts and EfficientNets.

Experimental results

Research questions

• RQ1 Can a normalization-free architecture paired with adaptive gradient clipping achieve competitive ImageNet accuracy and faster training than BN-based nets?
• RQ2 How do data augmentation and regularization interact with NFNet blocks to achieve state-of-the-art performance?
• RQ3 What architectural choices (depth distribution, width patterns, additional convolutions) yield the best training speed-accuracy trade-offs for NFNets?
• RQ4 Do NFNets retain advantages in transfer learning after large-scale pre-training compared to BN-based networks?
• RQ5 How does adaptive gradient clipping influence stability and performance for very large batch training?

Key findings

• NFNet-F1 matches EfficientNet-B7 accuracy while being 8.7x faster to train.
• Largest NFNet variants reach state-of-the-art top-1 accuracy of 86.5% with SAM.
• NFNets outperform BN-based networks when fine-tuned after pre-training on a 300M-image dataset, achieving 89.2% top-1.
• AGC enables stable training of NFNets with batch sizes up to 4096 and strong augmentations where NFResNets without AGC fail.
• NFNet-F5 reaches 86.0% top-1 on ImageNet, competitive with larger EfficientNet variants in FLOPs vs accuracy while prioritizing training latency.