[论文解读] Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization

David Peer, Bart J. Keulen+3 more|arXiv (Cornell University)|Aug 1, 2022

Advanced Neural Network Applications被引用 4

一句话总结

本论文提出逐层批量熵正则化，通过确保信息在每一层中保持正向流动，提升深度神经网络的可训练性，使无需跳跃连接、批量归一化或其他架构修改的500层‘原始’全连接和卷积网络也能成功训练。该方法在视觉和自然语言处理任务中，对残差网络、自编码器和变换器等多种架构均提升了性能。

ABSTRACT

Training deep neural networks is a very demanding task, especially challenging is how to adapt architectures to improve the performance of trained models. We can find that sometimes, shallow networks generalize better than deep networks, and the addition of more layers results in higher training and test errors. The deep residual learning framework addresses this degradation problem by adding skip connections to several neural network layers. It would at first seem counter-intuitive that such skip connections are needed to train deep networks successfully as the expressivity of a network would grow exponentially with depth. In this paper, we first analyze the flow of information through neural networks. We introduce and evaluate the batch-entropy which quantifies the flow of information through each layer of a neural network. We prove empirically and theoretically that a positive batch-entropy is required for gradient descent-based training approaches to optimize a given loss function successfully. Based on those insights, we introduce batch-entropy regularization to enable gradient descent-based training algorithms to optimize the flow of information through each hidden layer individually. With batch-entropy regularization, gradient descent optimizers can transform untrainable networks into trainable networks. We show empirically that we can therefore train a "vanilla" fully connected network and convolutional neural network -- no skip connections, batch normalization, dropout, or any other architectural tweak -- with 500 layers by simply adding the batch-entropy regularization term to the loss function. The effect of batch-entropy regularization is not only evaluated on vanilla neural networks, but also on residual networks, autoencoders, and also transformer models over a wide range of computer vision as well as natural language processing tasks.

研究动机与目标

• 解决深度神经网络中的退化问题，即随着深度增加，训练误差和测试误差反而上升。
• 从信息论角度探究深度网络不可训练的根本原因。
• 开发一种正则化方法，确保信息在每一层中正向流动，使原本不可训练的网络变得可训练。
• 在包括变换器和残差网络在内的多种架构与任务中，证明批量熵正则化的有效性。
• 在无需额外预训练的情况下，提升预训练模型（如BERT）的泛化能力和性能。

提出的方法

• 使用小批量统计量定义批量熵，作为衡量信息流经每一层的指标。
• 理论上证明：当任意层的批量熵为零时，梯度下降无法优化损失函数。
• 引入正则化项 ℒ_BE，惩罚低批量熵，确保训练过程中保持正向信息流动。
• 将 ℒ_BE 整合到整体损失函数中，联合优化任务性能与网络可学习性。
• 将该正则化方法应用于全连接、卷积、残差、自编码器和变换器架构。
• 使用网格搜索调优控制批量熵正则化强度的超参数 α。

实验结果

研究问题

• RQ1 为何极深的‘原始’神经网络尽管具有指数级表达能力，仍无法训练？
• RQ2 每一层的正批量熵是否为成功进行梯度下降优化的必要条件？
• RQ3 在不使用跳跃连接或批量归一化等架构修改的情况下，正则化批量熵是否能提升深度网络的可训练性？
• RQ4 批量熵正则化对不同架构（包括变换器和残差网络）的性能影响如何？
• RQ5 在无需额外预训练的情况下，批量熵正则化能否提升如BERT等预训练模型的性能？

主要发现

• 正批量熵是梯度下降能够优化损失函数的必要条件，因为批量熵为零会导致梯度消失，损失曲面不可优化。
• 批量熵正则化使无需跳跃连接、批量归一化或Dropout的500层全连接和卷积网络成功训练成为可能。
• 该方法在视觉和自然语言处理任务中均提升了性能，例如在使用 ℒ_BE 的 BERT Base 上，MPRC 指标相比未使用该正则化的 BERT Large 提升了 0.07 个百分点。
• 残差网络在较高 α 值（约 1.0）时收益最大，而变换器在较低 α 值（约 0.3）时表现更优，表明信息压缩有助于变换器的分类性能。
• 批量熵正则化可提升泛化能力，并可能增强对抗鲁棒性，但此方向仍需进一步研究。
• 该正则化方法在多种架构中均有效，包括自编码器和变换器，表明其适用范围远超标准前馈网络。