[Paper Review] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

Tianyi Chen, Tianyu Ding+3 more|arXiv (Cornell University)|Oct 24, 2023

Topic Modeling4 citations

TL;DR

LoRAShear proposes an efficient, limited-resource structured pruning framework for large language models (LLMs) that leverages LoRA module dependency graphs and a novel LoRA Half-Space Projected Gradient (LHSPG) optimizer to enable progressive, knowledge-preserving pruning. It further introduces dynamic knowledge recovery using adaptive pretraining and instruction fine-tuning data, achieving only 1.0% performance degradation at 20% model compression and 82% performance retention at 50% compression on LLAMA-v1.

ABSTRACT

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to structurally prune LLMs and recover knowledge. Given general LLMs, LoRAShear at first creates the dependency graphs over LoRA modules to discover minimally removal structures and analyze the knowledge distribution. It then proceeds progressive structured pruning on LoRA adaptors and enables inherent knowledge transfer to better preserve the information in the redundant structures. To recover the lost knowledge during pruning, LoRAShear meticulously studies and proposes a dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow down the performance gap to the full models. Numerical results demonstrate that by only using one GPU within a couple of GPU days, LoRAShear effectively reduced footprint of LLMs by 20% with only 1.0% performance degradation and significantly outperforms state-of-the-arts. The source code will be available at https://github.com/microsoft/lorashear.

Motivation & Objective

• To address the high computational cost and resource demands of pruning large language models (LLMs) in real-world deployment scenarios.
• To enable structured pruning of general LLMs without requiring full pretraining or access to original pretraining datasets.
• To preserve model knowledge during pruning by identifying and protecting critical, minimally removable structures in LoRA-adapted models.
• To recover lost general and domain-specific knowledge post-pruning through adaptive, multi-stage fine-tuning using curated datasets.
• To achieve state-of-the-art performance in pruned LLMs using only one GPU and a few GPU days.

Proposed method

• Constructs dependency graphs over LoRA modules to identify minimally removable structural units and partition trainable parameters accordingly.
• Introduces LoRA Half-Space Projected Gradient (LHSPG), a structured sparsity optimization method that transfers knowledge from redundant to critical LoRA structures during progressive pruning.
• Performs progressive structured pruning by iteratively removing low-saliency LoRA modules while preserving performance through knowledge transfer.
• Proposes a dynamic knowledge recovery mechanism that adaptively selects and uses subsets of pretraining data based on performance distribution to restore general knowledge.
• Combines the recovered general knowledge with standard instruction fine-tuning on datasets like Alpaca to restore instruction-following and domain-specific capabilities.
• Employs a two-stage fine-tuning protocol: first recovering general knowledge from curated pretraining data, then refining with instruction-tuning data.

Figure 1 : Overview of LoRAShear. Given a general LLM, LoRAShear at first discovers the minimally removal structures, then analyzes the knowledge distribution to mark the crucial ones as unprunable, then performs progressive structurally pruning over the prunable structures via LHSPG , and finally r

Experimental results

Research questions

• RQ1 Can structured pruning of LLMs be efficiently performed in a limited-resource setting without access to original pretraining data?
• RQ2 How can knowledge be preserved during structured pruning when pruning non-trainable LLM weights and auxiliary LoRA modules?
• RQ3 What is the impact of dynamic data selection from pretraining corpora on knowledge recovery in pruned LLMs?
• RQ4 To what extent can knowledge transfer via LHSPG reduce performance degradation during progressive pruning of LoRA-adapted LLMs?
• RQ5 Can a unified framework achieve both high compression and minimal performance loss in pruned LLMs using only one GPU?

Key findings

• At a 20% pruning ratio, LoRAShear achieves only 1.0% performance degradation compared to the full LLAMA-v1 model, significantly outperforming state-of-the-art methods.
• At a 50% pruning ratio, LoRAShear preserves 82% of the full model’s performance across multiple benchmarks, demonstrating strong robustness under high compression.
• LoRAShear outperforms existing methods such as LLM-Pruner, LoRAPrune, and WANDA by 2.2% to 5.0% in accuracy under the same 20% pruning ratio.
• The dynamic knowledge recovery stage, which combines curated pretraining data and instruction fine-tuning, is critical to closing the performance gap to the full model.
• The method achieves these results using only a single A100 GPU within a few GPU days, making it practical for public and resource-constrained users.
• Knowledge distribution analysis reveals that the first and last few LoRA node groups are most sensitive, justifying their exclusion from pruning.

(a) Dependency graph for MLP layers.