[Paper Review] A Deep Conditioning Treatment of Neural Networks

Naman Agarwal, Pranjal Awasthi+1 more|arXiv (Cornell University)|Feb 4, 2020

Adversarial Robustness in Machine Learning82 references4 citations

TL;DR

This paper proposes a deep conditioning treatment for neural networks by normalizing activations to improve data conditioning, showing that depth exponentially improves the condition number of kernel matrices toward 1. The method enables faster training and generalization in overparameterized networks, with theoretical guarantees for both top-layer and full-weight training, and empirically validates normalized ReLU (NormReLU) as a viable alternative to batch normalization.

ABSTRACT

We study the role of depth in training randomly initialized overparameterized neural networks. We give a general result showing that depth improves trainability of neural networks by improving the conditioning of certain kernel matrices of the input data. This result holds for arbitrary non-linear activation functions under a certain normalization. We provide versions of the result that hold for training just the top layer of the neural network, as well as for training all layers, via the neural tangent kernel. As applications of these general results, we provide a generalization of the results of Das et al. (2019) showing that learnability of deep random neural networks with a large class of non-linear activations degrades exponentially with depth. We also show how benign overfitting can occur in deep neural networks via the results of Bartlett et al. (2019b). We also give experimental evidence that normalized versions of ReLU are a viable alternative to more complex operations like Batch Normalization in training deep neural networks.

Motivation & Objective

• To understand how depth enhances the trainability of overparameterized neural networks through improved data conditioning.
• To formalize the role of activation normalization in reducing condition numbers of kernel matrices, regardless of activation type.
• To demonstrate that depth exponentially improves optimization and generalization by enhancing kernel matrix conditioning.
• To show that benign overfitting and learnability in random deep networks are enabled by this conditioning mechanism.
• To propose NormReLU as a practical alternative to batch normalization, validated through experiments on CIFAR-10.

Proposed method

• Normalizing activations so that, under standard Gaussian inputs, outputs have zero mean and unit variance, regardless of the base activation function.
• Defining a coefficient of non-linearity to quantify the rate at which condition numbers converge to 1 with depth.
• Analyzing the condition number of kernel matrices in both top-layer training and full-network training under the lazy training regime.
• Using the neural tangent kernel (NTK) framework to extend results to full-weight training with small learning rates.
• Introducing NormReLU as a normalized variant of ReLU, requiring no architectural changes.
• Conducting experiments on CIFAR-10 with fully connected and ResNet architectures to compare NormReLU against batch normalization, layer normalization, and SeLU.

Experimental results

Research questions

• RQ1 How does depth affect the conditioning of kernel matrices in randomly initialized deep neural networks?
• RQ2 Can activation normalization universally improve the trainability of deep networks across different activation functions?
• RQ3 Does depth lead to exponentially faster convergence in training error, independent of initial data conditioning?
• RQ4 Can benign overfitting occur in deep random neural networks due to improved conditioning?
• RQ5 Can a normalized ReLU variant like NormReLU replace batch normalization in training deep networks?

Key findings

• The condition number of kernel matrices converges to 1 exponentially fast in depth, with the rate governed by the activation's non-linearity coefficient.
• Gradient descent with square loss achieves ε-training error in O(log(1/ε)) iterations, regardless of initial data conditioning, due to improved conditioning from depth.
• Learning a target function implemented by a deep random neural network requires exponentially many queries in depth, even with constant probability over initialization.
• Minimum norm interpolating solutions in deep random networks can achieve non-trivial excess risk, demonstrating benign overfitting.
• NormReLU achieves test accuracy comparable to batch normalization and layer normalization on CIFAR-10, with slightly better performance in deeper networks.
• The normalized ReLU variant outperforms or matches SeLU in both fully connected and residual architectures, suggesting it as a viable alternative to batch normalization.