[论文解读] The Effects of Regularization and Data Augmentation are Class Dependent

Randall Balestriero, Léon Bottou+1 more|arXiv (Cornell University)|Apr 7, 2022

Machine Learning and Data Classification被引用 39

一句话总结

这篇论文表明，像数据增强和权重衰减等常见正则化方法会产生类别相关偏差，提升平均准确率的同时对某些类别的性能造成巨大下降，并将偏差转移到下游任务。

ABSTRACT

Regularization is a fundamental technique to prevent over-fitting and to improve generalization performances by constraining a model's complexity. Current Deep Networks heavily rely on regularizers such as Data-Augmentation (DA) or weight-decay, and employ structural risk minimization, i.e. cross-validation, to select the optimal regularization hyper-parameters. In this study, we demonstrate that techniques such as DA or weight decay produce a model with a reduced complexity that is unfair across classes. The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e.g. on Imagenet with a resnet50, the "barn spider" classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training. Even more surprising, such performance drop also appears when introducing uninformative regularization techniques such as weight decay. Those results demonstrate that our search for ever increasing generalization performance -- averaged over all classes and samples -- has left us with models and regularizers that silently sacrifice performances on some classes. This scenario can become dangerous when deploying a model on downstream tasks e.g. an Imagenet pre-trained resnet50 deployed on INaturalist sees its performances fall from $70\%$ to $30\%$ on class \#8889 when introducing random crop DA during the Imagenet pre-training phase. Those results demonstrate that designing novel regularizers without class-dependent bias remains an open research question.

研究动机与目标

• 激发并量化正则化如何在各类别之间塑造模型偏差。
• 证明数据增强对某些类别可保持标签不变，但对其他类别则不可保持。
• 显示权重衰减会引入与数据增强相似的类别相关偏差。
• 检验正则化偏差如何转移到下游的迁移学习任务。

提出的方法

• 提供理论直觉，说明当变换不能保持真实标签的水平集时，数据增强如何引入不可约的偏差。
• 在 ImageNet 上跨多种架构，基于不同数据增强强度的逐类别性能进行经验分析。
• 通过对不同 DA 策略训练模型并对类别精度进行统计检验，量化逐类别偏差的敏感性分析。
• 通过权重衰减重复分析，展示对于无信息正则化器的逐类别偏差。
• 通过变化的 DA 进行预训练并在目标数据集（INaturalist）上评估逐类别性能，探讨迁移学习。

实验结果

研究问题

• RQ1 数据增强在提高总体平均准确度的同时，是否引入类别相关偏差？
• RQ2 观察到的逐类别偏差是否在不同架构和数据集上保持一致？
• RQ3 权重衰减是否产生与数据增强相似的类别相关偏差？
• RQ4 在预训练阶段的正则化是否会在迁移学习中对下游任务产生偏置？
• RQ5 从源数据集向目标数据集转移的类别相关偏差有多大？

主要发现

• 数据增强可以提升平均测试准确率，但对某些类别的逐类准确率降低。
• 来自 DA 的逐类别偏差是类别相关的，并且在不同架构和不同 DA 形式（随机裁剪、CutOut、颜色抖动）下都存在。
• 权重衰减同样会引入类别相关偏差，即使回归器是无信息的。
• 正则化带来的偏差在迁移学习场景下可能转移到下游任务，影响目标类别的性能。
• 在源数据集上按平均性能选择的模型，可能在目标数据集中感兴趣的类别上具有最大的偏见。