[论文解读] Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning

Yihua Zhang, Yimeng Zhang+7 more|arXiv (Cornell University)|Oct 13, 2023

Domain Adaptation and Few-Shot LearningComputer Science被引用 3

一句话总结

本文提出了两种新颖的训练集剪枝方法——标签映射（LM）与特征映射（FM）——用于迁移学习，可在不损失性能的前提下实现高达80%的源数据集剪枝。通过利用源域与目标域之间的对齐，该方法使预训练速度提升2–5倍，同时在多种任务和模型上保持下游性能。

ABSTRACT

Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged as an effective way to improve data efficiency by identifying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e., how to prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy on downstream target tasks. To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning as separate problems. By contrast, we establish a unified viewpoint to integrate DP with transfer learning and find that existing DP methods are not suitable for the transfer learning paradigm. We then propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-target domain mapping. Furthermore, we demonstrate the effectiveness of our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to 40% ~ 80% without sacrificing downstream performance, resulting in a significant 2 ~ 5 times speed-up during the pretraining stage. Besides, our proposal exhibits broad applicability and can improve other computationally intensive transfer learning techniques, such as adversarial pretraining. Codes are available at https://github.com/OPTML-Group/DP4TL.

研究动机与目标

• 为解决迁移学习中数据集剪枝的空白，现有方法在域内剪枝时因源域与目标域的间接影响而失效。
• 开发适用于迁移学习的高效、低开销剪枝技术，涵盖监督学习与自监督预训练。
• 在保持或提升下游微调准确率的同时，实现显著的预训练加速。
• 通过源域-目标域映射，建立统一框架，将数据集剪枝与迁移学习联系起来。
• 在多种架构、数据集和预训练范式（包括对抗性预训练）中展示方法的广泛适用性。

提出的方法

• 标签映射（LM）基于小规模代理模型，根据源类别与目标类别的语义和特征层面的对齐程度，选择源类别。
• 特征映射（FM）将该方法扩展至自监督预训练，通过匹配源域与目标域之间的特征表示实现剪枝。
• 两种方法均使用小型快速代理模型（如ResNet-18）估算源类别对下游性能的影响，避免完整模型的重新训练。
• 剪枝过程由基于相关性评分的函数指导，按源类别对目标任务的相关性进行排序，从而最小化计算成本。
• 该框架对最终主干模型保持无偏，支持在不同架构和预训练目标间复用。
• 通过ε-锐度和Hessian范数等指标分析损失景观的平坦度，验证剪枝数据带来的泛化能力提升。

Figure 1 : Left : An illustration of the proposed dataset pruning methods (LM and FM) and their performance overview. Large scale source dataset is pruned by LM and FM through a small surrogate model (ResNet-18). Large foundation models can achieve up to $5\times$ speed-up on pretraining without no

实验结果

研究问题

• RQ1 数据集剪枝能否有效扩展至源域与目标域不同的迁移学习范式？
• RQ2 为何传统域内数据集剪枝方法在迁移学习场景中失效？
• RQ3 我们能否设计一种低复杂度、可扩展的方法，以识别并移除对迁移学习有害或冗余的源数据类别？
• RQ4 在不降低下游性能的前提下，源数据最多可剪枝至何种程度，适用于多样化的任务与模型？
• RQ5 基于源域-目标域对齐的剪枝，如何影响下游任务中损失景观的泛化性与平坦度？

主要发现

• 所提出的标签映射（LM）与特征映射（FM）方法在ImageNet上实现高达80%的剪枝，且在OxfordPets与StanfordCars等下游任务上无性能损失。
• 在ResNet-101等大模型上，预训练速度提升2–5倍，且在多个基准测试中下游准确率无下降。
• 在40%的剪枝比例下，LM在大多数迁移学习任务（包括少样本设置）中成功识别出“胜出子集”。
• 剪枝后的源数据集使下游任务的损失景观更加平坦，通过ε-锐度、Hessian范数和Fisher-Rao范数度量，表明泛化能力提升。
• 通过t-SNE进行的特征可视化表明，保留的源类别在语义和空间上更接近目标类别，而被剪枝的类别则分散且远离目标。
• 该方法具有良好的泛化性：即使使用不同代理模型（如VGG、ResNet-32s），在80%剪枝比例下，所选源类别与默认的ResNet-18基线的重叠度高达97.7%。

Figure 3 : Preliminary studies on the usage of surrogate models for LM . ( a ) The downstream performance (on Flowers102) of using the source model ResNet-101 trained on the pruned ImageNet delivered by LM at different pruning ratios. Here LM is conducted using either RN-101 or a smaller surrogate m