[论文解读] Data Dropout: Optimizing Training Data for Convolutional Neural Networks

Tianyang Wang, Jun Huan+1 more|arXiv (Cornell University)|Sep 1, 2018

Advanced Neural Network Applications参考文献 21被引用 15

一句话总结

本文提出数据丢弃（Data Dropout）方法，一种两轮训练策略，通过识别并移除不利的训练样本，提升深度学习模型的泛化能力。通过测量每个样本对验证损失的影响，该方法可减少训练集规模，从而在不进行微调或增加额外训练开销的前提下，提升ResNet和DnCNN等CNN模型在图像分类与去噪任务中的性能。

ABSTRACT

Deep learning models learn to fit training data while they are highly expected to generalize well to testing data. Most works aim at finding such models by creatively designing architectures and fine-tuning parameters. To adapt to particular tasks, hand-crafted information such as image prior has also been incorporated into end-to-end learning. However, very little progress has been made on investigating how an individual training sample will influence the generalization ability of a model. In other words, to achieve high generalization accuracy, do we really need all the samples in a training dataset? In this paper, we demonstrate that deep learning models such as convolutional neural networks may not favor all training samples, and generalization accuracy can be further improved by dropping those unfavorable samples. Specifically, the influence of removing a training sample is quantifiable, and we propose a Two-Round Training approach, aiming to achieve higher generalization accuracy. We locate unfavorable samples after the first round of training, and then retrain the model from scratch with the reduced training dataset in the second round. Since our approach is essentially different from fine-tuning or further training, the computational cost should not be a concern. Our extensive experimental results indicate that, with identical settings, the proposed approach can boost performance of the well-known networks on both high-level computer vision problems such as image classification, and low-level vision problems such as image denoising.

研究动机与目标

• 探究移除特定训练样本是否能提升深度学习模型的泛化性能。
• 开发一种识别并消除不利训练样本的方法，以优化训练数据集。
• 设计一种可泛化的训练框架，在不修改模型架构或微调的前提下，提升现有CNN模型的性能。
• 在包括高阶（分类）和低阶（去噪）任务在内的多样化计算机视觉任务中，证明数据优化的有效性。

提出的方法

• 第一轮在完整训练集上训练CNN，以计算每个训练样本对验证损失的影响。
• 对每个训练样本，通过测量其从验证集中移除后验证损失的变化，来估计其影响。
• 识别并移除影响值为正的样本——即移除后能降低整体验证误差的样本。
• 通过排除不利样本，重构一个更小但更优的训练集。
• 在第二轮中，从头开始在缩减后的数据集上重新训练模型，确保不进行微调或继续训练。
• 使用标准优化方法（Adam）和训练协议，仅在第二轮中应用数据增强，以保证公平性。

实验结果

研究问题

• RQ1 移除特定训练样本是否能提升已训练CNN的泛化准确率？
• RQ2 如何对单个训练样本对模型性能的影响进行定量测量？
• RQ3 所提出的两轮训练方法是否在不修改模型架构的前提下，优于标准训练方法？
• RQ4 该数据丢弃方法是否可在不同CNN架构和计算机视觉任务中实现泛化？
• RQ5 性能提升是源于数据优化步骤，还是仅仅因为额外的训练轮次？

主要发现

• 所提出的两轮训练方法结合数据丢弃，可在图像分类与图像去噪任务中均提升泛化准确率。
• 在BSD68数据集上，该方法使DnCNN的平均PSNR提升了约0.04 dB，表明在图像去噪任务中实现了可测量的性能提升。
• 该方法在视觉效果和定量指标上均优于原始DnCNN模型，甚至优于广泛使用的去噪方法BM3D。
• 该方法在多种模型（包括ResNet、DenseNet和All-CNN-C）上均表现有效，显示出广泛的适用性。
• 第一轮模型仅用于计算影响值；第二轮训练从头开始，因此与微调或继续训练有本质区别。
• 尽管训练样本数量减少，第二轮模型仍取得更优性能，表明在某些情况下，数据质量比数量更为关键。