[Paper Review] Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

Jonathan Frankle, David J. Schwab+1 more|arXiv (Cornell University)|Feb 29, 2020

Domain Adaptation and Few-Shot Learning36 references79 citations

TL;DR

The paper shows that training only BatchNorm affine parameters (gamma and beta) with all other weights frozen yields surprisingly high accuracy, demonstrating strong expressive power of per-feature affine transformations on random features across deep CNNs.

ABSTRACT

A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.

Motivation & Objective

• Assess the expressive power of BatchNorm affine parameters (gamma and beta) when all other network weights are frozen at initialization.
• Quantify how well networks trained only BatchNorm perform on CIFAR-10 and ImageNet compared to fully trained networks.
• Investigate how depth and width affect performance when only BatchNorm is trainable.
• Analyze how gamma and beta values evolve and contribute to feature pruning and sparsity.

Proposed method

• Freeze all network weights at random initialization except BatchNorm affine parameters (gamma and beta) which are trained.
• Evaluate networks across varying depths and widths of ResNets on CIFAR-10 and ImageNet.
• Compare performance to fully trained networks and to training an equivalent number of randomly selected parameters.
• Analyze distributions of learned gamma/beta and their effect on feature sparsity and activation sparsity.

Experimental results

Research questions

• RQ1 How expressive are per-feature BatchNorm parameters when trained alone on random features?
• RQ2 What accuracy can be achieved on CIFAR-10 and ImageNet by training only gamma and beta across deep CNNs?
• RQ3 How do network depth and width influence performance under this restricted training regime?
• RQ4 Do gamma/beta learn to disable a subset of features, and how does this affect activations?

Key findings

• Training only gamma and beta yields high accuracy relative to training random subsets of parameters (e.g., CIFAR-10 up to 82% with deep networks; ImageNet top-5 up to 32%).
• Equivalently sized random parameters perform far worse than BatchNorm affine parameters, highlighting per-feature power of gamma and beta.
• Gamma learns to suppress roughly a quarter to a third of channels (values near zero) during BatchNorm-only training, indicating per-feature sparsity.
• Deeper and wider networks improve BatchNorm-only accuracy, with depth contributing more than width for a given BatchNorm parameter budget.
• When outputs are trained alongside BatchNorm, accuracy improves further, implying affine parameters are crucial but not solely sufficient for SOTA performance.
• Activations are sparsified under BatchNorm-only training, with a notable fraction of features effectively disabled by gamma near zero.