[Paper Review] Fixup Initialization: Residual Learning Without Normalization

Hongyi Zhang, Yann Dauphin+1 more|arXiv (Cornell University)|Jan 27, 2019

Advanced Neural Network Applications29 references112 citations

TL;DR

Fixup initialization enables training very deep residual networks without normalization by carefully scaling residual branches, biases, and multipliers so updates are properly scaled; it matches normalization performance on several tasks and can achieve state-of-the-art results with proper regularization.

ABSTRACT

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Motivation & Objective

• Motivate the need to train deep residual networks without normalization by analyzing exploding gradients at initialization.
• Propose Fixup initialization to scale residual branches for stable, maximal-rate training without normalization.
• Empirically evaluate Fixup on image classification and machine translation benchmarks against normalization baselines.
• Demonstrate that with proper regularization Fixup can achieve competitive or state-of-the-art results on standard datasets.

Proposed method

• Analyze gradient behavior of unnormalized ResNets and derive a lower bound on gradient norms under positive homogeneity.
• Introduce Fixup initialization by (a) rescaling residual branch weights by L^{-1/(2m-2)}, (b) initializing the last layer of each residual branch and the classifier layer to zero, (c) adding a scalar multiplier in every branch, and (d) adding a scalar bias before each convolution/linear/activation layer.
• Explain the role of biases and multipliers in stabilizing training and mimicking normalization dynamics without actual normalization.
• Provide practical guidelines and Ablation insights through experiments on CIFAR-10, ImageNet, and machine translation tasks.

Experimental results

Research questions

• RQ1 Can a deep residual network be trained reliably without normalization?
• RQ2 If so, can it be trained with the same maximal learning rate and achieve comparable generalization?
• RQ3 What initialization and architectural adjustments (Fixup) are necessary to stabilize training and match normalization-based baselines across tasks?

Key findings

• Fixup enables training very deep residual networks without normalization at maximal learning rate, matching BatchNorm performance in depth studies up to 10,000 layers.
• On CIFAR-10, Fixup with proper regularization improves test error relative to standard Xavier initialization and can achieve competitive results without normalization.
• On ImageNet, Fixup improves training stability and test error over standard initialization, with Mixup regularization helping Close the gap to normalization-based methods.
• In machine translation, replacing LayerNorm with Fixup yields BLEU scores that match or exceed the LayerNorm baseline, demonstrating cross-domain effectiveness.
• Biases and a per-branch scalar multiplier help emulate normalization-like dynamics and stabilize learning without computing statistics.