[Paper Review] Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training

Geng Yuan, Yanyu Li+6 more|arXiv (Cornell University)|Sep 22, 2022

Machine Learning and ELM4 citations

TL;DR

This paper proposes SpFDE, a generic framework for sparse training that integrates layer freezing and data sieving to reduce training costs without sacrificing accuracy. By progressively freezing early layers and dynamically shrinking the training dataset, SpFDE achieves significant reductions in FLOPs and memory usage while outperforming state-of-the-art methods on ImageNet and CIFAR-100 at various sparsity levels.

ABSTRACT

Recently, sparse training has emerged as a promising paradigm for efficient deep learning on edge devices. The current research mainly devotes efforts to reducing training costs by further increasing model sparsity. However, increasing sparsity is not always ideal since it will inevitably introduce severe accuracy degradation at an extremely high sparsity level. This paper intends to explore other possible directions to effectively and efficiently reduce sparse training costs while preserving accuracy. To this end, we investigate two techniques, namely, layer freezing and data sieving. First, the layer freezing approach has shown its success in dense model training and fine-tuning, yet it has never been adopted in the sparse training domain. Nevertheless, the unique characteristics of sparse training may hinder the incorporation of layer freezing techniques. Therefore, we analyze the feasibility and potentiality of using the layer freezing technique in sparse training and find it has the potential to save considerable training costs. Second, we propose a data sieving method for dataset-efficient training, which further reduces training costs by ensuring only a partial dataset is used throughout the entire training process. We show that both techniques can be well incorporated into the sparse training algorithm to form a generic framework, which we dub SpFDE. Our extensive experiments demonstrate that SpFDE can significantly reduce training costs while preserving accuracy from three dimensions: weight sparsity, layer freezing, and dataset sieving.

Motivation & Objective

• To explore alternative cost-reduction strategies in sparse training beyond increasing sparsity, which often leads to accuracy degradation.
• To investigate the feasibility and effectiveness of layer freezing in sparse training, a technique previously unexplored in this domain.
• To develop a data sieving method that enables end-to-end dataset-efficient training by dynamically selecting informative samples.
• To integrate layer freezing and data sieving into a generic, plug-and-play framework for sparse training that maintains high accuracy.
• To demonstrate that combining weight sparsity, layer freezing, and dataset reduction yields superior training efficiency and memory savings.

Proposed method

• Proposes a progressive layer freezing strategy that identifies and freezes early layers in sparse models based on structural and representational similarity during training.
• Introduces a dynamic data sieving technique that continuously filters and retains only the most informative samples from the dataset throughout training.
• Designs a generic framework, SpFDE, that seamlessly integrates weight sparsity, layer freezing, and data sieving into a unified training pipeline.
• Employs a training FLOPs-based metric to evaluate cost reduction, accounting for sparsity, frozen layers, and reduced dataset size.
• Uses a 32-bit floating-point representation with batch size 64 to measure memory costs, comparing SpFDE against baseline DST methods.
• Applies ablation studies to isolate the contributions of layer freezing and data sieving, validating their individual and combined effects.

Experimental results

Research questions

• RQ1 Can layer freezing be effectively applied in sparse training despite the dynamic sparsity pattern of Dynamic Sparse Training (DST)?
• RQ2 How does data sieving impact model accuracy and training efficiency in sparse training, especially compared to static dataset shrinking?
• RQ3 What is the combined impact of weight sparsity, layer freezing, and dataset reduction on training FLOPs and memory costs?
• RQ4 How does SpFDE compare to state-of-the-art sparse training methods in terms of accuracy and efficiency across different sparsity levels?
• RQ5 What is the optimal trade-off between accuracy, FLOPs, memory usage, and actual training acceleration when combining these three techniques?

Key findings

• SpFDE achieves higher top-1 accuracy than MEST and other SOTA methods at the same FLOPs level on ImageNet using ResNet-50, with the best result of 76.03% at 0.95×10^18 FLOPs.
• At 95% sparsity, SpFDE reduces training FLOPs by up to 43.9% compared to baseline DST methods, with minimum memory costs significantly lower than existing approaches.
• The average memory cost of SpFDE is 20–25.3% lower than the minimum required by baseline DST methods, even when accounting for additional overheads like periodic dense back-propagation.
• Data sieving in SpFDE dynamically updates the training set, mitigating overfitting and resulting in higher accuracy than one-time dataset shrinking used in MEST.
• Ablation studies confirm that both layer freezing and data sieving independently contribute to performance gains, with their combination yielding the strongest results.
• SpFDE outperforms RigL-ITOP, the previous SOTA, with 76.03% top-1 accuracy at 0.95×10^18 FLOPs, while reducing memory usage and FLOPs compared to baseline methods.