[Paper Review] Evolving Normalization-Activation Layers

Hanxiao Liu, Andrew Brock+2 more|arXiv (Cornell University)|Apr 6, 2020

Advanced Neural Network Applications67 references54 citations

TL;DR

The paper unifies normalization and activation into a single search space and uses evolution with rejection protocols and multi-architecture evaluation to discover EvoNorms, novel normalization-activation layers that generalize across CNN backbones and tasks.

ABSTRACT

Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated approach. Instead of designing them separately, we unify them into a single tensor-to-tensor computation graph, and evolve its structure starting from basic mathematical functions. Examples of such mathematical functions are addition, multiplication and statistical moments. The use of low-level mathematical functions, in contrast to the use of high-level modules in mainstream NAS, leads to a highly sparse and large search space which can be challenging for search methods. To address the challenge, we develop efficient rejection protocols to quickly filter out candidate layers that do not work well. We also use multi-objective evolution to optimize each layer's performance across many architectures to prevent overfitting. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures that go beyond existing design patterns. For example, some EvoNorms do not assume that normalization and activation functions must be applied sequentially, nor need to center the feature maps, nor require explicit activation functions. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets but also transfer well to Mask R-CNN with FPN/SpineNet for instance segmentation and to BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers in many cases.

Motivation & Objective

• Motivate the joint design of normalization and activation layers rather than treating them separately.
• Formulate a tensor-to-tensor computation graph using low-level mathematical operations to search for new building blocks.
• Develop rejection protocols and multi-objective evolution to handle sparse search spaces and promote cross-architecture generalization.
• Discover EvoNorms and demonstrate their effectiveness across image classification, instance segmentation, and GAN training.

Proposed method

• Unify normalization and activation into a single computation graph built from primitive operations (addition, multiplication, moments, etc.).
• Represent layers as DAGs with a fixed node budget and 14 total nodes including inputs and trainable constants.
• Use random graph generation and three-step mutation to explore the space during evolution.
• Apply two rejection protocols (quality and stability) to prune poor or unstable layers early.
• Evaluate candidates across multiple anchor architectures (ResNet-50, MobileNetV2, EfficientNet-B0) to enforce cross-architecture generalization.
• Perform multi-objective tournament selection (average vs Pareto frontier) to guide evolution.
• Re-rank top layers on ImageNet and validate on COCO (Mask R-CNN) and BigGAN for cross-domain transfer.

Experimental results

Research questions

• RQ1 Can normalization and activation be effectively discovered jointly from low-level mathematical operations?
• RQ2 Do EvoNorms generalize across diverse architectures and tasks beyond classification?
• RQ3 What search strategies (rejection protocols and multi-architecture evaluation) are effective for sparse, high-dimensional NAS spaces?

Key findings

• EvoNorms consistently outperform BN-ReLU across multiple architectures on ImageNet (ResNet, MobileNetV2, EfficientNet).
• Batch-dependent EvoNorms (B-series) show strong gains on Mask R-CNN COCO instance segmentation over BN-ReLU and GN-ReLU combinations.
• Batch-independent EvoNorms (S-series) achieve competitive or better results than GN-ReLU and FRN across varying batch sizes, with robust performance when batch statistics are unavailable.
• EvoNorm-B0 combines batch and instance statistics in its denominator and can forgo an explicit activation function, illustrating nonstandard but effective design patterns.
• EvoNorms transfer effectively to non-classification tasks: improved Mask R-CNN AP metrics, and competitive BigGAN IS/FID results, indicating strong generalization.
• Analysis reveals EvoNorms often promote scale-invariance and include mixed variances and tensor-to-tensor transformations, suggesting design principles beyond traditional normalization.