[论文解读] The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

George Philipp|arXiv (Cornell University)|May 25, 2021

Adversarial Robustness in Machine Learning参考文献 217被引用 4

一句话总结

本文提出了非线性系数（NLC），这是一种新颖的标量度量，用于在训练前量化神经网络架构的非线性程度。通过基于随机初始化网络的NLC高精度预测测试误差，作者实现了零样本架构设计——使实践者能够在无需训练的情况下，利用理论基础坚实、计算高效的度量，优化模型复杂度与性能。

ABSTRACT

In essence, a neural network is an arbitrary differentiable, parametrized function. Choosing a neural network architecture for any task is as complex as searching the space of those functions. For the last few years, 'neural architecture design' has been largely synonymous with 'neural architecture search' (NAS), i.e. brute-force, large-scale search. NAS has yielded significant gains on practical tasks. However, NAS methods end up searching for a local optimum in architecture space in a small neighborhood around architectures that often go back decades, based on CNN or LSTM. In this work, we present a different and complementary approach to architecture design, which we term 'zero-shot architecture design' (ZSAD). We develop methods that can predict, without any training, whether an architecture will achieve a relatively high test or training error on a task after training. We then go on to explain the error in terms of the architecture definition itself and develop tools for modifying the architecture based on this explanation. This confers an unprecedented level of control on the deep learning practitioner. They can make informed design decisions before the first line of code is written, even for tasks for which no prior art exists. Our first major contribution is to show that the 'degree of nonlinearity' of a neural architecture is a key causal driver behind its performance, and a primary aspect of the architecture's model complexity. We introduce the 'nonlinearity coefficient' (NLC), a scalar metric for measuring nonlinearity. Via extensive empirical study, we show that the value of the NLC in the architecture's randomly initialized state before training is a powerful predictor of test error after training and that attaining a right-sized NLC is essential for attaining an optimal test error. The NLC is also conceptually simple, well-defined for any feedforward network, easy and cheap to compute, has extensive theoretical, empirical and conceptual grounding, follows instructively from the architecture definition, and can be easily controlled via our 'nonlinearity normalization' algorithm. We argue that the NLC is the most powerful scalar statistic for architecture design specifically and neural network analysis in general. Our analysis is fueled by mean field theory, which we use to uncover the 'meta-distribution' of layers. Beyond the NLC, we uncover and flesh out a range of metrics and properties that have a significant explanatory influence on test and training error. We go on to explain the majority of the error variation across a wide range of randomly generated architectures with these metrics and properties. We compile our insights into a practical guide for architecture designers, which we argue can significantly shorten the trial-and-error phase of deep learning deployment. Our results are grounded in an experimental protocol that exceeds that of the vast majority of other deep learning studies in terms of carefulness and rigor. We study the impact of e.g. dataset, learning rate, floating-point precision, loss function, statistical estimation error and batch inter-dependency on performance and other key properties. We promote research practices that we believe can significantly accelerate progress in architecture design research.

研究动机与目标

• 开发一种无需训练即可预测神经网络性能的方法，从而在编写代码前就能做出明智的架构决策。
• 识别非线性性作为模型复杂度与泛化性能的主要驱动因素。
• 提出一种理论基础坚实、计算高效的标量度量——非线性系数（NLC），用于量化架构的非线性程度。
• 提供一种实用且可解释的神经架构设计框架，减少对昂贵神经架构搜索（NAS）的依赖。
• 建立严格的实验协议，以隔离并分析关键超参数与统计因素对性能的影响。

提出的方法

• 非线性系数（NLC）从架构的权重矩阵和激活函数在随机初始化状态下的公式推导得出，该公式源自场论均值。
• NLC 衡量了跨层非线性变换的期望幅度，捕捉了架构内在的非线性表征能力。
• 引入了一种“非线性性归一化”算法，用于将NLC调整至目标值，从而实现受控的架构修改。
• 该方法利用场论均值推导出层激活的“元分布”，为NLC提供了理论基础。
• 在大量随机生成的架构上进行了经验验证，性能在多个数据集和超参数设置下进行测量。
• 通过统计分析，隔离了数据集、学习率、精度、损失函数和批量间依赖性对误差和NLC的影响。

实验结果

研究问题

• RQ1 在训练前测量的神经网络架构的非线性性，能否预测其最终的测试误差？
• RQ2 非线性系数（NLC）是否在各种架构和任务中均为模型性能的可靠且稳定预测指标？
• RQ3 通过归一化过程调整NLC，能在多大程度上控制架构性能？
• RQ4 学习率、批量大小和浮点精度等因素如何影响NLC与泛化误差之间的关系？
• RQ5 NLC及相关度量能否解释在广泛随机生成的架构空间中训练误差与测试误差的大部分方差？

主要发现

• 在随机初始化状态下计算的非线性系数（NLC）是最终测试误差的强预测指标，在多种架构中均表现出高度相关性。
• NLC值处于“合适大小”范围内的架构显著降低了测试误差，表明最优非线性性对性能至关重要。
• NLC定义明确，计算高效，并在场论均值理论基础上具有坚实的理论支撑，为架构分析提供了原则性基础。
• 非线性性归一化算法使NLC的精确控制成为可能，使实践者能够在不训练的情况下调节模型复杂度。
• NLC及相关度量在广泛随机生成的架构中，解释了训练误差与测试误差变化的显著部分。
• 研究发现，学习率、批量间依赖性及浮点精度等因素显著影响性能，但其影响受NLC的调节。