[Paper Review] The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

George Philipp|arXiv (Cornell University)|May 25, 2021

Adversarial Robustness in Machine Learning217 references4 citations

TL;DR

This paper introduces the Nonlinearity Coefficient (NLC), a novel scalar metric that quantifies the degree of nonlinearity in neural network architectures before training. By predicting test error with high accuracy based on the NLC of randomly initialized networks, the authors enable zero-shot architecture design—allowing practitioners to optimize model complexity and performance without training, using a theoretically grounded, computationally efficient metric grounded in mean field theory.

ABSTRACT

In essence, a neural network is an arbitrary differentiable, parametrized function. Choosing a neural network architecture for any task is as complex as searching the space of those functions. For the last few years, 'neural architecture design' has been largely synonymous with 'neural architecture search' (NAS), i.e. brute-force, large-scale search. NAS has yielded significant gains on practical tasks. However, NAS methods end up searching for a local optimum in architecture space in a small neighborhood around architectures that often go back decades, based on CNN or LSTM. In this work, we present a different and complementary approach to architecture design, which we term 'zero-shot architecture design' (ZSAD). We develop methods that can predict, without any training, whether an architecture will achieve a relatively high test or training error on a task after training. We then go on to explain the error in terms of the architecture definition itself and develop tools for modifying the architecture based on this explanation. This confers an unprecedented level of control on the deep learning practitioner. They can make informed design decisions before the first line of code is written, even for tasks for which no prior art exists. Our first major contribution is to show that the 'degree of nonlinearity' of a neural architecture is a key causal driver behind its performance, and a primary aspect of the architecture's model complexity. We introduce the 'nonlinearity coefficient' (NLC), a scalar metric for measuring nonlinearity. Via extensive empirical study, we show that the value of the NLC in the architecture's randomly initialized state before training is a powerful predictor of test error after training and that attaining a right-sized NLC is essential for attaining an optimal test error. The NLC is also conceptually simple, well-defined for any feedforward network, easy and cheap to compute, has extensive theoretical, empirical and conceptual grounding, follows instructively from the architecture definition, and can be easily controlled via our 'nonlinearity normalization' algorithm. We argue that the NLC is the most powerful scalar statistic for architecture design specifically and neural network analysis in general. Our analysis is fueled by mean field theory, which we use to uncover the 'meta-distribution' of layers. Beyond the NLC, we uncover and flesh out a range of metrics and properties that have a significant explanatory influence on test and training error. We go on to explain the majority of the error variation across a wide range of randomly generated architectures with these metrics and properties. We compile our insights into a practical guide for architecture designers, which we argue can significantly shorten the trial-and-error phase of deep learning deployment. Our results are grounded in an experimental protocol that exceeds that of the vast majority of other deep learning studies in terms of carefulness and rigor. We study the impact of e.g. dataset, learning rate, floating-point precision, loss function, statistical estimation error and batch inter-dependency on performance and other key properties. We promote research practices that we believe can significantly accelerate progress in architecture design research.

Motivation & Objective

• To develop a method for predicting neural network performance without training, enabling informed architecture decisions before code is written.
• To identify nonlinearity as a primary driver of model complexity and generalization performance.
• To introduce a theoretically grounded, computationally efficient scalar metric—The Nonlinearity Coefficient (NLC)—that quantifies architectural nonlinearity.
• To provide a practical, explainable framework for neural architecture design that reduces reliance on costly neural architecture search (NAS).
• To establish a rigorous experimental protocol that isolates and analyzes the impact of key hyperparameters and statistical factors on performance.

Proposed method

• The Nonlinearity Coefficient (NLC) is computed from the architecture's weight matrices and activation functions in the randomly initialized state, using a formula derived from mean field theory.
• The NLC measures the expected magnitude of nonlinear transformations across layers, capturing the architecture's intrinsic capacity for nonlinear representation.
• A 'nonlinearity normalization' algorithm is introduced to adjust the NLC to a target value, enabling controlled architectural modifications.
• The method leverages mean field theory to derive the 'meta-distribution' of layer activations, providing theoretical grounding for the NLC.
• Empirical validation is conducted across a wide range of randomly generated architectures, with performance measured across multiple datasets and hyperparameter settings.
• Statistical analysis isolates the influence of dataset, learning rate, precision, loss function, and batch inter-dependency on error and NLC.

Experimental results

Research questions

• RQ1 Can the nonlinearity of a neural network architecture, measured before training, predict its eventual test error?
• RQ2 Is the Nonlinearity Coefficient (NLC) a reliable and stable predictor of model performance across diverse architectures and tasks?
• RQ3 To what extent can architectural performance be controlled by adjusting the NLC via a normalization procedure?
• RQ4 How do factors like learning rate, batch size, and floating-point precision affect the relationship between NLC and generalization error?
• RQ5 Can the NLC and related metrics explain the majority of variance in training and test error across a broad space of randomly generated architectures?

Key findings

• The Nonlinearity Coefficient (NLC) computed in the randomly initialized state is a strong predictor of final test error, with high correlation across diverse architectures.
• Architectures with an NLC value in a 'right-sized' range achieve significantly lower test error, indicating that optimal nonlinearity is crucial for performance.
• The NLC is well-defined, computationally efficient, and theoretically grounded in mean field theory, providing a principled basis for architectural analysis.
• The nonlinearity normalization algorithm enables precise control over the NLC, allowing practitioners to tune model complexity without training.
• The NLC and related metrics explain a substantial portion of the variation in training and test error across a wide range of randomly generated architectures.
• The study reveals that factors such as learning rate, batch inter-dependency, and floating-point precision significantly influence performance, but their effects are modulated by the NLC.