[论文解读] The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter

Ajay Jaiswal, Shiwei Liu+2 more|arXiv (Cornell University)|Jun 6, 2023

Domain Adaptation and Few-Shot Learning被引用 6

一句话总结

本文提出了'本质稀疏性'——一种在大规模预训练Transformer模型中出现的现象，即在一次性剪枝中移除最小幅度权重时，当稀疏度超过临界阈值后，模型性能会急剧下降。作者表明，高质量的稀疏子网络存在于密集检查点中而无需微调，从而实现无需重新训练的免费、可迁移的剪枝，即使在现代大语言模型（如Vicuna-7B）中也成立。

ABSTRACT

Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of repetitive train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. This paper comprehensively studies induced sparse patterns across multiple large pre-trained vision and language transformers. We propose the existence of -- essential sparsity defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in one-shot without re-training. We also find essential sparsity to hold valid for N:M sparsity patterns as well as on modern-scale large language models (Vicuna-7B). We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a counter-intuitive finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at \url{https://github.com/VITA-Group/essential_sparsity}.

研究动机与目标

• 识别并验证大规模预训练Transformer中一种此前被忽视的结构性特性——'本质稀疏性'。
• 证明通过一次性剪枝（即移除最小幅度权重）可识别出高质量的稀疏子网络，而无需重新训练。
• 探究本质稀疏性是否在包括现代大语言模型（如Vicuna-7B）在内的多种架构中成立。
• 探索在BERT中剪枝特性出现的机制，特别是其在预训练过程中的演变，以及其对预训练数据和损失目标的依赖性。
• 比较一次性剪枝（OMP）与迭代剪枝（LTH）的性能，并评估其在下游任务中的可迁移性。

提出的方法

• 提出'本质稀疏性'作为一次性剪枝中性能急剧下降的临界稀疏度阈值。
• 通过移除最小幅度权重对预训练检查点执行一次性剪枝（OMP），不进行任何微调或重新训练。
• 使用剪枝后的子网络在下游任务上评估性能，将OMP与彩票理论（LTH）和SparseGPT进行比较。
• 在BERT上进行消融实验，分析预训练过程中剪枝特性的出现，通过改变数据量和预训练目标。
• 使用LTH与OMP剪枝掩码之间的余弦相似度来评估掩码相似性及任务间的可迁移性。
• 在Vicuna-7B上使用MMLU基准评估零样本性能，验证现代大语言模型中的本质稀疏性。

Figure 1: Naturally induced sparsity patterns of bert-base-uncased across the components of transformer blocks. The pre-trained model is pruned by 21.50% using one-shot-magnitude pruning. Yellow dots indicate the location of pruned low-magnitude weights.

实验结果

研究问题

• RQ1 本质稀疏性是否存在于大规模预训练视觉与语言Transformer中？其在下游任务中是否具有可迁移性？
• RQ2 一次性剪枝能否识别出与通过迭代训练（LTH）发现的子网络性能相当的高质量稀疏子网络？
• RQ3 BERT的预训练过程中是否存在剪枝特性的突变式出现？若存在，何时以及为何发生？
• RQ4 预训练数据量如何影响模型将知识压缩到更少参数的能力？
• RQ5 自监督学习目标是否比监督学习目标引发更强的涌现稀疏性？

主要发现

• 本质稀疏性存在于多种大规模预训练模型中，包括ViT、BERT和Vicuna-7B，且在超过临界稀疏度阈值后性能出现急剧下降。
• 一次性剪枝（OMP）在稀疏度低于50%时性能与LTH相当，表明高质量稀疏子网络存在于密集检查点中而无需重新训练。
• OMP与LTH掩码在本质稀疏性范围内具有较高的余弦相似度，表明OMP以零额外成本识别出相似且可迁移的子网络。
• BERT在预训练过程中表现出突变式稀疏化，经过一定训练迭代后迅速变得高度稀疏，表明其结构具有动态涌现特性。
• 在更大数据集上预训练的模型展现出更强的知识压缩能力，表明其参数效率更高。
• 自监督学习（SSL）目标比监督学习（SL）目标引发更强的涌现稀疏性，表明SSL更有利于内在稀疏性的形成。

Figure 2: Fine-tuning performance drop estimated with respect to dense counterpart for various downstream tasks of NLP pre-trained models ( bert-base, OPT-125m, OPT-350m, OPT-1.3B ). Note that for fair evaluation, we have used exactly same fine-tuning settings across all pruning ratios.