[论文解读] NoisyNN: Exploring the Impact of Information Entropy Change in Learning Systems

Xiaowei Yu, Huang, Zhe+4 more|arXiv (Cornell University)|Sep 19, 2023

Neural Networks and Applications被引用 5

一句话总结

该论文提出了NoisyNN，一种通过主动向深度学习模型注入正向噪声以降低信息熵、从而减少任务复杂度的框架，实现了最先进性能。通过将噪声分类为正向（降低熵）和有害类型，作者证明了受控的噪声注入可提升准确率——在ImageNet上实现超过95%的top-1准确率，并在域自适应任务中取得最先进结果，挑战了噪声普遍具有破坏性的传统观点。

ABSTRACT

We investigate the impact of entropy change in deep learning systems by noise injection at different levels, including the embedding space and the image. The series of models that employ our methodology are collectively known as Noisy Neural Networks (NoisyNN), with examples such as NoisyViT and NoisyCNN. Noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different learning tasks like image classification and transfer learning. However, this work shows noise can be an effective way to change the entropy of the learning system. We demonstrate that specific noise can boost the performance of various deep models under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the task complexity. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieved an unprecedented top 1 accuracy of 95$\%$ on ImageNet. Both theoretical analysis and empirical evidence have confirmed that the presence of positive noise, can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us that we can influence the performance of learning systems via information entropy change.

研究动机与目标

• 通过信息熵的视角重新评估噪声的作用，挑战噪声普遍损害深度学习模型的常规假设。
• 探究是否可战略性地利用噪声以降低任务复杂度并提升模型泛化能力。
• 开发一种系统性框架——NoisyNN，用于对噪声类型（正向与有害）进行分类并加以利用以提升性能。
• 通过实证验证正向噪声在多种架构（CNNs、ViTs）和任务（图像分类、域自适应）中的优势。
• 为将噪声作为调控信息熵、提升学习效率的工具提供理论与实证依据。

提出的方法

• 噪声在两个层面注入：输入图像空间和深度模型的潜在空间表征中。
• 研究了三种噪声类型：高斯噪声、线性变换噪声和椒盐噪声，系统性地应用以评估其影响。
• 理论分析将性能提升与通过信息熵最小化降低任务复杂度相联系。
• 正向噪声（PN）定义为能降低任务熵的噪声；有害噪声（HN）则增加熵，依据此熵标准进行区分。
• NoisyNN框架在训练过程中应用噪声注入，且为每种模型和任务调优超参数。
• 实验采用标准基准（ImageNet、Office-Home、VisDA2017），使用ViT-B和ResNet-50主干网络，并与最先进方法进行对比。

实验结果

研究问题

• RQ1 通过降低信息熵，噪声注入是否能减少任务复杂度并提升模型性能？
• RQ2 在何种条件下，噪声在深度学习系统中发挥积极作用而非造成损害？
• RQ3 不同类型的噪声（高斯、线性变换、椒盐）如何影响CNN和视觉Transformer的熵与性能？
• RQ4 正向噪声注入是否能在图像分类和域自适应任务中实现最先进结果？
• RQ5 支持熵降低作为性能提升机制的理论与实证依据是什么？

主要发现

• 使用NoisyTVT-B，在ImageNet上实现了93.5%的top-1准确率，超越了以往最先进模型。
• 在VisDA2017基准上，NoisyTVT-B实现了90.0%的平均准确率，优于现有方法（包括CDTrans-B和SSRT-B）。
• 在Office-Home的域自适应任务中，NoisyTVT-B实现了90.0%的平均准确率，超过此前最先进水平88.8%（SSRT-B）。
• 理论分析证实，正向噪声通过降低信息熵来减少任务复杂度，为性能提升提供了原则性依据。
• 实证结果表明，正向噪声在多个基准上均一致提升了CNN和ViT架构的性能。
• 本研究证明，噪声并非普遍具有破坏性——其影响取决于是否降低或增加任务熵，从而重新定义了其在深度学习中的角色。