[Paper Review] Really should we pruning after model be totally trained? Pruning based on a small amount of training.

Yue Li, Weibin Zhao+1 more|arXiv (Cornell University)|Jan 24, 2019

Advanced Neural Network Applications33 references4 citations

TL;DR

This paper proposes Incremental Pruning based on Less Training (IPLT), a novel pruning algorithm that achieves competitive model compression and acceleration with minimal pre-training—only a few epochs—by incrementally pruning during training. IPLT enables 8x–9x model compression and up to 10x speedup in both training and inference for VGG-19 on CIFAR-10, challenging the assumption that extensive pre-training is necessary for effective pruning.

ABSTRACT

Pre-training of models in pruning algorithms plays an important role in pruning decision-making. We find that excessive pre-training is not necessary for pruning algorithms. According to this idea, we propose a pruning algorithm---Incremental pruning based on less training (IPLT). Compared with the traditional pruning algorithm based on a large number of pre-training, IPLT has competitive compression effect than the traditional pruning algorithm under the same simple pruning strategy. On the premise of ensuring accuracy, IPLT can achieve 8x-9x compression for VGG-19 on CIFAR-10 and only needs to pre-train few epochs. For VGG-19 on CIFAR-10, we can not only achieve 10 times test acceleration, but also about 10 times training acceleration. At present, the research mainly focuses on the compression and acceleration in the application stage of the model, while the compression and acceleration in the training stage are few. We newly proposed a pruning algorithm that can compress and accelerate in the training stage. It is novel to consider the amount of pre-training required by pruning algorithm. Our results have implications: Too much pre-training may be not necessary for pruning algorithms.

Motivation & Objective

• To challenge the assumption that extensive pre-training is essential for effective neural network pruning.
• To develop a pruning algorithm that enables model compression and acceleration during the training phase.
• To minimize the amount of pre-training required while maintaining high accuracy and compression efficiency.
• To explore whether reduced pre-training can still yield competitive pruning performance compared to traditional methods.

Proposed method

• IPLT introduces an incremental pruning strategy that applies pruning in stages during training, rather than after full pre-training.
• The algorithm performs pruning based on a small number of initial training epochs, significantly reducing the pre-training phase.
• It uses a simple pruning strategy that maintains model accuracy while progressively removing redundant parameters.
• The method integrates pruning directly into the training loop, enabling simultaneous compression and acceleration during model optimization.
• It leverages weight magnitude-based pruning decisions, but applies them incrementally after minimal pre-training.
• The approach is designed to be compatible with standard training pipelines, minimizing architectural or training procedure changes.

Experimental results

Research questions

• RQ1 Can effective neural network pruning be achieved with significantly reduced pre-training compared to conventional methods?
• RQ2 Does incremental pruning during training outperform or match traditional pruning that relies on extensive pre-training?
• RQ3 What is the minimal amount of pre-training required to achieve competitive compression and acceleration in pruning?
• RQ4 Can pruning be effectively integrated into the training process to accelerate both training and inference?
• RQ5 Does reducing pre-training lead to a trade-off in model accuracy or compression efficiency?

Key findings

• IPLT achieves 8x–9x model compression for VGG-19 on CIFAR-10 using only a few epochs of pre-training, matching or exceeding the performance of traditional methods.
• The method enables up to 10x acceleration in both training and inference, significantly reducing computational cost.
• Despite minimal pre-training, IPLT maintains high test accuracy, demonstrating that extensive pre-training is not necessary for effective pruning.
• The algorithm shows that pruning can be effectively applied during training, enabling compression and acceleration from the outset.
• The results challenge the prevailing assumption that long pre-training phases are essential for successful pruning, suggesting that pre-training can be drastically reduced.