[Paper Review] Self-Normalizing Neural Networks

Günter Klambauer, Thomas Unterthiner+2 more|arXiv (Cornell University)|Jun 8, 2017

Machine Learning in Materials Science515 citations

TL;DR

The paper introduces Self-Normalizing Neural Networks (SNNs) based on scaled ELU activations (SELUs) and a specific weight initialization to automatically drive activations toward zero mean and unit variance, enabling very deep networks without batch normalization. It provides theoretical guarantees (fixed-point stability and bounded variance) and demonstrates superior performance across multiple benchmarks compared to standard FNNs and normalization-based methods.

ABSTRACT

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

Motivation & Objective

• Motivate the need for deep, robust feed-forward networks that can learn rich representations without heavy normalization tricks.
• Propose a self-normalizing mechanism via SELU activations and a specific weight initialization to maintain stable activation statistics across layers.
• Provide theoretical guarantees (fixed-point convergence, variance bounds) for self-normalization.
• Demonstrate empirical superiority of SNNs over various normalization schemes and competing models on diverse datasets.

Proposed method

• Introduce SELU activation: selu(x) = lambda x for x>0; lambda*(alpha e^x - alpha) for x<=0, with alpha and lambda chosen to achieve a stable fixed point.
• Define the mapping g from layer activation mean/variance (mu, nu) to the next layer's (mu~, nu~) via moments of the SELU-transformed Gaussian z ~ N(mu*omega, sqrt(nu*tau)); derive analytical expressions for mu~ and nu~ (Equations 4 and 5).
• Set weight initialization with omega = 0 and tau = 1 for normalized fixed-point behavior; prove that (mu, nu) converges to a stable fixed point, typically (0,1), under SELU parameters (alpha_01, lambda_01).
• Prove via Banach fixed-point theorem that g is a contraction on a domain Omega, ensuring a unique attracting fixed point and self-normalization.
• Establish variance bounds (Theorems 2 and 3) to prevent exploding/vanishing gradients, showing nu stays within a controllable range across many layers.
• Introduce alpha-dropout, an adaptation of dropout that preserves mean/variance for SELUs to maintain self-normalization during training.

Experimental results

Research questions

• RQ1 Can SELU activations with a specific initialization induce self-normalization across many network layers?
• RQ2 Do self-normalizing properties prevent vanishing/exploding gradients and allow deeper FNNs to train robustly?
• RQ3 How do SNNs perform compared to batch normalization, layer normalization, weight normalization, Highway/ResNet in diverse benchmarks?
• RQ4 What empirical gains do very deep SNNs achieve on UCI tasks, drug discovery (Tox21), and astronomy datasets?

Key findings

• SNNs significantly outperform competing FNNs on 121 UCI tasks in pairwise comparisons.
• On Tox21, deeper SNNs (up to 8 layers) surpass shallow batchnorm/weightnorm networks and set a new benchmark with 8-layer SNNs achieving top performance.
• In astronomy (HTRU2 pulsar dataset), SNNs achieve state-of-the-art AUC of 0.98, surpassing Naive Bayes, C4.5, and SVM baselines.
• SNNs tend to use much deeper architectures (average depth ≈ 10.8 layers) than competitors to achieve best accuracies.
• Theoretical results establish a stable, attracting fixed point for mean/variance (0,1) under normalized weights and provide variance bounds that prevent exploding/vanishing gradients.