[论文解读] Self-Normalizing Neural Networks

Günter Klambauer, Thomas Unterthiner+2 more|arXiv (Cornell University)|Jun 8, 2017

Machine Learning in Materials Science被引用 515

一句话总结

本文提出基于缩放ELU激活（SELUs）和特定权重初始化的自归一化神经网络（SNNs），能够在没有批归一化的情况下实现很深的网络，自动将激活推向零均值和单位方差。它给出理论保证（固定点稳定性和有界方差）并在多个基准测试中相对于标准前馈神经网络和归一化方法显示出更优的性能。

ABSTRACT

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

研究动机与目标

• Motivate the need for deep, robust feed-forward networks that can learn rich representations without heavy normalization tricks.
• Propose a self-normalizing mechanism via SELU activations and a specific weight initialization to maintain stable activation statistics across layers.
• Provide theoretical guarantees (fixed-point convergence, variance bounds) for self-normalization.
• Demonstrate empirical superiority of SNNs over various normalization schemes and competing models on diverse datasets.

提出的方法

• Introduce SELU activation: selu(x) = lambda x for x>0; lambda*(alpha e^x - alpha) for x<=0, with alpha and lambda chosen to achieve a stable fixed point.
• Define the mapping g from layer activation mean/variance (mu, nu) to the next layer's (mu~, nu~) via moments of the SELU-transformed Gaussian z ~ N(mu*omega, sqrt(nu*tau)); derive analytical expressions for mu~ and nu~ (Equations 4 and 5).
• Set weight initialization with omega = 0 and tau = 1 for normalized fixed-point behavior; prove that (mu, nu) converges to a stable fixed point, typically (0,1), under SELU parameters (alpha_01, lambda_01).
• Prove via Banach fixed-point theorem that g is a contraction on a domain Omega, ensuring a unique attracting fixed point and self-normalization.
• Establish variance bounds (Theorems 2 and 3) to prevent exploding/vanishing gradients, showing nu stays within a controllable range across many layers.
• Introduce alpha-dropout, an adaptation of dropout that preserves mean/variance for SELUs to maintain self-normalization during training.

实验结果

研究问题

• RQ1 Can SELU activations with a specific initialization induce self-normalization across many network layers?
• RQ2 Do self-normalizing properties prevent vanishing/exploding gradients and allow deeper FNNs to train robustly?
• RQ3 How do SNNs perform compared to batch normalization, layer normalization, weight normalization, Highway/ResNet in diverse benchmarks?
• RQ4 What empirical gains do very deep SNNs achieve on UCI tasks, drug discovery (Tox21), and astronomy datasets?

主要发现

• SNNs significantly outperform competing FNNs on 121 UCI tasks in pairwise comparisons.
• On Tox21, deeper SNNs (up to 8 layers) surpass shallow batchnorm/weightnorm networks and set a new benchmark with 8-layer SNNs achieving top performance.
• In astronomy (HTRU2 pulsar dataset), SNNs achieve state-of-the-art AUC of 0.98, surpassing Naive Bayes, C4.5, and SVM baselines.
• SNNs tend to use much deeper architectures (average depth ≈ 10.8 layers) than competitors to achieve best accuracies.
• Theoretical results establish a stable, attracting fixed point for mean/variance (0,1) under normalized weights and provide variance bounds that prevent exploding/vanishing gradients.