[Paper Review] E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity

Yunwei Li, Lin Niu+4 more|arXiv (Cornell University)|Oct 24, 2023

Topic Modeling4 citations

TL;DR

E-Sparse proposes a one-shot, entropy-based N:M sparsity method for large language models (LLMs) that uses information entropy of hidden state features to guide channel-level pruning, combined with global and local channel shuffling to preserve model accuracy. It achieves up to 1.53× faster inference and 43.52% memory savings on LLaMA and OPT models with minimal accuracy loss, leveraging NVIDIA Ampere GPUs via a custom Sparse-GEMM implementation in FasterTransformer.

ABSTRACT

Traditional pruning methods are known to be challenging to work in Large Language Models (LLMs) for Generative AI because of their unaffordable training process and large computational demands. For the first time, we introduce the information entropy of hidden state features into a pruning metric design, namely E-Sparse, to improve the accuracy of N:M sparsity on LLM. E-Sparse employs the information richness to leverage the channel importance, and further incorporates several novel techniques to put it into effect: (1) it introduces information entropy to enhance the significance of parameter weights and input feature norms as a novel pruning metric, and performs N:M sparsity without modifying the remaining weights. (2) it designs global naive shuffle and local block shuffle to quickly optimize the information distribution and adequately cope with the impact of N:M sparsity on LLMs' accuracy. E-Sparse is implemented as a Sparse-GEMM on FasterTransformer and runs on NVIDIA Ampere GPUs. Extensive experiments on the LLaMA family and OPT models show that E-Sparse can significantly speed up the model inference over the dense model (up to 1.53X) and obtain significant memory saving (up to 43.52%), with acceptable accuracy loss.

Motivation & Objective

• Address the high computational and memory costs of deploying large language models (LLMs) in real-world generative AI applications.
• Overcome the limitations of traditional pruning methods in LLMs, which often require expensive retraining and fail to leverage hardware-optimized sparsity patterns.
• Design a post-training, one-shot pruning method that enables efficient N:M sparsity without fine-tuning, suitable for rapid deployment on modern GPUs.
• Improve the accuracy of N:M sparsity in LLMs by introducing information entropy as a novel metric for channel importance, capturing the richness of hidden state features.
• Develop a channel shuffling strategy to mitigate accuracy degradation caused by concentrated information distribution across channels during pruning.

Proposed method

• Introduce a novel pruning metric that combines information entropy of hidden state features with input feature norms and parameter magnitudes to assess channel importance.
• Use information entropy to quantify the information richness within each channel of the hidden states, enabling more accurate identification of critical parameters for retention.
• Apply a global naive shuffle and a local block shuffle to redistribute information across channels, reducing concentration and preserving performance after N:M pruning.
• Implement a custom Sparse-GEMM kernel optimized for N:M sparsity in FasterTransformer, natively supporting NVIDIA Ampere GPUs with Sparse Tensor Core acceleration.
• Perform one-shot pruning without retraining or weight updates, relying solely on the entropy-informed metric and shuffling techniques to maintain accuracy.
• Integrate the method into the inference pipeline with minimal overhead, enabling efficient deployment on standard LLM inference workloads.

Experimental results

Research questions

• RQ1 Can information entropy of hidden state features serve as a more effective metric than magnitude or norm alone for guiding N:M sparsity in LLMs?
• RQ2 How does the concentration of information across channels affect the accuracy of N:M structured pruning in LLMs?
• RQ3 Can channel shuffling techniques—both global and local—effectively mitigate accuracy degradation caused by structured pruning in LLMs?
• RQ4 To what extent can entropy-based N:M sparsity accelerate inference and reduce memory usage in LLMs without retraining?
• RQ5 How does the proposed method compare to existing post-training pruning approaches like Wanda or SparseGPT in terms of speed, memory, and accuracy?

Key findings

• E-Sparse achieves up to 1.53× faster inference compared to the dense FP16 baseline on LLaMA and OPT models, with consistent speedups across both context and decoder stages.
• The method reduces memory usage by up to 43.52% on LLaMA models, with larger models showing proportionally greater memory savings.
• E-Sparse improves perplexity by 1.32 points over state-of-the-art methods on LLaMA, demonstrating superior accuracy retention despite aggressive pruning.
• The combination of entropy-based pruning and channel shuffling reduces accuracy loss by preserving information-rich channels during N:M sparsity.
• The custom Sparse-GEMM kernel in FasterTransformer enables efficient inference on NVIDIA Ampere GPUs, with up to 34.8% latency reduction on A100 hardware.
• The method maintains strong performance across different model sizes and sequence lengths, showing robustness and generalization in real-world deployment scenarios.