[Paper Review] MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Geng Yuan, Xiaolong Ma+14 more|arXiv (Cornell University)|Oct 26, 2021

Advanced Neural Network ApplicationsComputer Science53 references41 citations

TL;DR

MEST introduces Elastic Mutation (EM) and Soft Memory Bound (EM&S) for memory-efficient, accurate sparse training on edge devices, with data-efficient two-phase training to remove less informative examples.

ABSTRACT

Recently, a new trend of exploring sparsity for accelerating neural network training has emerged, embracing the paradigm of training on the edge. This paper proposes a novel Memory-Economic Sparse Training (MEST) framework targeting for accurate and fast execution on edge devices. The proposed MEST framework consists of enhancements by Elastic Mutation (EM) and Soft Memory Bound (&S) that ensure superior accuracy at high sparsity ratios. Different from the existing works for sparse training, this current work reveals the importance of sparsity schemes on the performance of sparse training in terms of accuracy as well as training speed on real edge devices. On top of that, the paper proposes to employ data efficiency for further acceleration of sparse training. Our results suggest that unforgettable examples can be identified in-situ even during the dynamic exploration of sparsity masks in the sparse training process, and therefore can be removed for further training speedup on edge devices. Comparing with state-of-the-art (SOTA) works on accuracy, our MEST increases Top-1 accuracy significantly on ImageNet when using the same unstructured sparsity scheme. Systematical evaluation on accuracy, training speed, and memory footprint are conducted, where the proposed MEST framework consistently outperforms representative SOTA works. A reviewer strongly against our work based on his false assumptions and misunderstandings. On top of the previous submission, we employ data efficiency for further acceleration of sparse training. And we explore the impact of model sparsity, sparsity schemes, and sparse training algorithms on the number of removable training examples. Our codes are publicly available at: https://github.com/boone891214/MEST.

Motivation & Objective

• Motivate edge-friendly sparse training to reduce memory, compute, and energy while preserving accuracy.
• Develop a sparse training framework (MEST) that maintains memory economy under edge constraints.
• Investigate how sparsity schemes influence accuracy, memory footprint, and training speed on real devices.
• Explore data efficiency in sparse training to further accelerate training without harming final accuracy.

Proposed method

• Define a sparse training framework (MEST) that maintains target sparsity s by mutations.
• Introduce Elastic Mutation (EM) to gradually reduce mutation rate and widen the search space without destabilizing convergence.
• Propose Soft Memory Bound (EM&S) to allow temporary growth and training of newly grown weights while keeping end sparsity, enabling memory flexibility.
• Compute an importance score combining weight magnitude and gradient magnitude to guide removal/growth of weights.
• Ensure mutation and sparsity operations are performed on indices to keep the process strictly sparse, preserving memory efficiency.
• Evaluate different sparsity schemes (unstructured, block, pattern) on edge devices to measure actual training speed, accuracy, and memory usage.
• Propose a data-efficient two-phase training (DE) to identify and remove less informative examples during training, using forgetting events as a criterion.

Experimental results

Research questions

• RQ1 How does the MEST framework perform in accuracy, training speed, and memory footprint across sparsity schemes on edge devices?
• RQ2 What is the impact of Elastic Mutation and Soft Memory Bound on end-model accuracy at high sparsity ratios?
• RQ3 Can data efficiency (two-phase training) further accelerate sparse training on the edge without harming final accuracy?
• RQ4 How do different sparsity schemes (unstructured, block, pattern) affect training speed and memory consumption on mobile hardware?

Key findings

• MEST+EM and especially MEST+EM&S consistently improve accuracy across sparsity ratios compared to SOTA sparse training methods on CIFAR-10/100 with ResNet-32/VGG-19.
• MEST achieves substantial training acceleration on edge devices, with up to ~2.3x gains for block/pattern schemes and notable memory footprint reductions, depending on scheme.
• Soft Memory Bound (EM&S) yields higher accuracy at extremely high sparsity (e.g., 98%) by allowing growth of newly formed weights before pruning, acting as an undo mechanism.
• Data-efficient two-phase training (DE) can further reduce training time by 10-22% in CIFAR-10/100 experiments without sacrificing final accuracy, leveraging unforgettable example removal.
• Across ImageNet with ResNet-50, MEST variants outperform comparable methods in accuracy under similar or lower FLOPs, while preserving memory efficiency on edge hardware.