[Paper Review] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models

Peijie Dong, Lujun Li+5 more|arXiv (Cornell University)|Jun 5, 2024

Natural Language Processing Techniques4 citations

TL;DR

Pruner-Zero introduces a genetic programming-based framework to automatically evolve symbolic pruning metrics for large language models without retraining. By searching over a curated space of weight, gradient, and activation operations, it discovers high-performing, human-free metrics that outperform SOTA post-training pruning methods on LLaMA and LLaMA-2 across language modeling and zero-shot tasks at 50% sparsity.

ABSTRACT

Despite the remarkable capabilities, Large Language Models (LLMs) face deployment challenges due to their extensive size. Pruning methods drop a subset of weights to accelerate, but many of them require retraining, which is prohibitively expensive and computationally demanding. Recently, post-training pruning approaches introduced novel metrics, enabling the pruning of LLMs without retraining. However, these metrics require the involvement of human experts and tedious trial and error. To efficiently identify superior pruning metrics, we develop an automatic framework for searching symbolic pruning metrics using genetic programming. In particular, we devise an elaborate search space encompassing the existing pruning metrics to discover the potential symbolic pruning metric. We propose an opposing operation simplification strategy to increase the diversity of the population. In this way, Pruner-Zero allows auto-generation of symbolic pruning metrics. Based on the searched results, we explore the correlation between pruning metrics and performance after pruning and summarize some principles. Extensive experiments on LLaMA and LLaMA-2 on language modeling and zero-shot tasks demonstrate that our Pruner-Zero obtains superior performance than SOTA post-training pruning methods. Code at: \url{https://github.com/pprp/Pruner-Zero}.

Motivation & Objective

• To address the high human effort and trial-and-error required in designing effective pruning metrics for large language models.
• To eliminate reliance on expert knowledge and retraining by automating the discovery of optimal pruning metrics.
• To explore the structural and functional correlations between pruning metrics and post-pruning model performance.
• To develop a scalable, format-robust, and diverse search space for symbolic metric evolution in post-training pruning.
• To enable efficient, high-performance model compression for LLMs through automated, symbolic metric generation.

Proposed method

• A genetic programming framework is designed to evolve symbolic pruning metrics from a predefined search space of operations and inputs.
• The search space includes weight (W), gradient (G), and activation (X) as inputs, with unary (UOP) and binary (BOP) operations such as absolute value, multiplication, min-max scaling, and element-wise functions.
• An opposing operation simplification strategy is introduced to enhance population diversity and avoid premature convergence.
• The framework evaluates candidate metrics using validation loss on calibration data, guiding evolution toward high-performing metrics.
• Symbolic expressions are evolved through selection, crossover, and mutation, with fitness determined by post-pruning model performance.
• The final metric is derived from the best-performing individual in the evolutionary process, which is then applied to prune LLMs without fine-tuning.

Figure 1: Overview of the Automatic Symbolic Pruning Metric Discovery Process in our Pruner-Zero framework. This process employs genetic programming to iteratively generate and refine symbolic pruning metrics via tournament selection, subtree crossover, and node mutation. Upon generating offspring,

Experimental results

Research questions

• RQ1 Can an automated framework discover superior pruning metrics for LLMs without human-designed heuristics?
• RQ2 How do different combinations of weight, gradient, and activation operations influence pruning performance?
• RQ3 What structural properties of symbolic metrics correlate with high post-pruning model accuracy?
• RQ4 Can genetic programming effectively explore and optimize complex, non-linear pruning metrics in a high-dimensional space?
• RQ5 How does the evolved metric compare to existing SOTA post-training pruning methods in terms of accuracy and sparsity?

Key findings

• Pruner-Zero discovers a symbolic pruning metric: ||W| × |W|| × σ(|G|), which achieves state-of-the-art performance on LLaMA and LLaMA-2 at 50% sparsity.
• The evolved metric outperforms existing post-training pruning methods, including SparseGPT, Wanda, and GBLM-Pruner, on both language modeling and zero-shot tasks.
• The framework identifies that metrics combining weight magnitude and gradient magnitude with normalization (σ) yield superior performance, suggesting a strong correlation between gradient sensitivity and pruning effectiveness.
• The opposing operation simplification strategy increases population diversity and improves convergence, leading to more robust metric discovery.
• The search process reveals that format sensitivity is reduced when using a comprehensive operation set, validating the design of the search space.
• Extensive ablation shows that the evolved metric maintains strong performance across multiple datasets and tasks, confirming its generalization capability.

Figure 2: Comparison between Evolution Search and Random Search Processes. Notably, the individual perplexity is lower in the evolution search method, leading to significant improvements in the search efficiency and overall stability.