[论文解读] Efficient Lottery Ticket Finding: Less Data is More

Zhenyu Zhang, Xuxi Chen+2 more|arXiv (Cornell University)|Jun 6, 2021

Video Analysis and Summarization被引用 7

一句话总结

本文提出Pruning-Aware Critical（PrAC）集合——一种小型、数据高效的子集，能够捕捉最困难且最具信息量的样本，用于彩票票券发现。通过在PrAC集合而非完整数据集上训练和剪枝密集网络，该方法在保持竞争力准确率的同时，将训练迭代次数减少高达92.77%，显著提升了发现胜出票券的效率。

ABSTRACT

The lottery ticket hypothesis (LTH) reveals the existence of winning tickets (sparse but critical subnetworks) for dense networks, that can be trained in isolation from random initialization to match the latter's accuracies. However, finding winning tickets requires burdensome computations in the train-prune-retrain process, especially on large-scale datasets (e.g., ImageNet), restricting their practical benefits. This paper explores a new perspective on finding lottery tickets more efficiently, by doing so only with a specially selected subset of data, called Pruning-Aware Critical set (PrAC set), rather than using the full training set. The concept of PrAC set was inspired by the recent observation, that deep networks have samples that are either hard to memorize during training, or easy to forget during pruning. A PrAC set is thus hypothesized to capture those most challenging and informative examples for the dense model. We observe that a high-quality winning ticket can be found with training and pruning the dense network on the very compact PrAC set, which can substantially save training iterations for the ticket finding process. Extensive experiments validate our proposal across diverse datasets and network architectures. Specifically, on CIFAR-10, CIFAR-100, and Tiny ImageNet, we locate effective PrAC sets at 35.32%~78.19% of their training set sizes. On top of them, we can obtain the same competitive winning tickets for the corresponding dense networks, yet saving up to 82.85%~92.77%, 63.54%~74.92%, and 76.14%~86.56% training iterations, respectively. Crucially, we show that a PrAC set found is reusable across different network architectures, which can amortize the extra cost of finding PrAC sets, yielding a practical regime for efficient lottery ticket finding.

研究动机与目标

• 解决在大规模数据集上通过迭代剪枝（IMP）寻找彩票票券所面临的高计算成本问题。
• 探究是否可以将数据稀疏化（核心子集选择）与模型稀疏化（胜出票券发现）协同设计，以提升效率。
• 识别出最小但最关键的训练数据子集，以保持稀疏子网络的泛化能力和可训练性。
• 开发一种可重用、紧凑的数据子集，实现在多种架构和数据集上高效、高质量的彩票票券识别。

提出的方法

• 将Pruning-Aware Critical（PrAC）集合定义为一种数据子集，能够捕捉在训练过程中最难记忆、在剪枝过程中最容易被遗忘的样本。
• 通过在早期训练期间使用激活统计量测量模型的记忆能力，识别出关键训练样本（CET）。
• 通过在迭代剪枝过程中使用基于梯度的显著性分数测量特征敏感度，识别出关键剪枝样本（CEP）。
• 将CET和CEP的并集作为PrAC集合，确保其包含训练和剪枝阶段均至关重要的样本。
• 在PrAC集合上训练时应用早停策略，进一步减少训练迭代次数而不损失性能。
• 在不同网络架构之间重用同一PrAC集合，以分摊子集识别的成本并提升实用性。

实验结果

研究问题

• RQ1 在彩票票券框架中，哪些训练样本对发现可训练的稀疏子网络（即胜出票券）最为关键？
• RQ2 是否可以构建一个紧凑且数据高效的子集，以保留发现胜出票券所需的可训练性和泛化能力？
• RQ3 将数据稀疏化（核心子集选择）与模型稀疏化（剪枝）协同设计，如何提升彩票票券发现的效率？
• RQ4 PrAC集合是否可跨不同架构重用而不会导致显著性能下降，从而实现可扩展且实用的部署？
• RQ5 通过早停和基于子集的训练，能在多大程度上减少训练迭代次数，同时在彩票票券搜索中保持竞争力的准确率？

主要发现

• 在CIFAR-10、CIFAR-100和Tiny ImageNet上，PrAC集合仅占完整训练集的35.32%至78.19%，却能成功发现高质量的胜出票券。
• 与全量数据训练相比，该方法在Tiny ImageNet上将训练迭代次数减少高达92.77%，在CIFAR-10上减少82.85%，在CIFAR-100上减少74.92%。
• PrAC彩票票券在性能上与原始彩票票券相当或更优，尤其在高稀疏度下表现更佳（例如，在CIFAR-10上96.48%稀疏度时优于原始票券）。
• PrAC集合可在不同架构（如ResNet-20和ResNet-56）之间重用，实现成本分摊与实际部署。
• 结合早停策略后，基于PrAC的票券发现可节省高达83.93%的训练资源，且准确率损失最多仅为0.50%。
• PrAC集合包含更多模糊、复杂的图像，具有杂乱背景和多个物体，表明其捕捉了难以学习且信息量高的样本。