[Paper Review] Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Zeyu Han, Chao Gao+3 more|arXiv (Cornell University)|Mar 21, 2024

Parallel Computing and Optimization Techniques90 citations

TL;DR

This paper surveys parameter-efficient fine-tuning (PEFT) methods for large models, categorizing algorithms, analyzing performance and system costs, and outlining real-world applications across modalities.

ABSTRACT

Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed ......

Motivation & Objective

• Motivate the need to adapt large pre-trained models to downstream tasks efficiently.
• Systematically categorize PEFT algorithms and their core mechanisms.
• Evaluate computational overheads and practical system implications of PEFT methods.
• Highlight applications of PEFT across NLP, vision, and multimodal models and discuss deployment considerations.

Proposed method

• Classify PEFT methods into additive, selective, reparameterized, and hybrid categories.
• Detail representative algorithms within each category (e.g., adapters, soft prompts, pruning, LoRA and its derivatives).
• Describe how these methods modify or leverage model parameters to achieve efficiency.
• Analyze techniques for reducing computation cost (KV-cache management, pruning, quantization, memory optimization).
• Discuss architecture- and system-level considerations for deploying PEFT (distributed tuning, query serving, concurrent tuning).
• Survey applications across LLMs, Vision Transformers, vision-language models, and diffusion models.

Experimental results

Research questions

• RQ1 What are the main PEFT algorithm families and their characteristic mechanisms?
• RQ2 How do additive, selective, reparameterized, and hybrid PEFT methods compare in terms of parameter efficiency and performance?
• RQ3 What are the practical system costs and deployment considerations for PEFT across different model families and tasks?
• RQ4 What are the key application domains and model architectures where PEFT is most impactful?

Key findings

• PEFT methods are categorized into four broad families: additive, selective, reparameterized, and hybrid, each with distinct design trade-offs.
• Adapters, soft prompts, and other additive techniques offer parameter-efficient alternatives to full fine-tuning, with varying efficiency and accuracy profiles.
• Selective PEFT uses masks or structured pruning to fine-tune subsets of parameters, improving hardware efficiency and scalability.
• Reparameterized PEFT, especially LoRA and its variants, achieves strong efficiency by learning low-rank updates while preserving fast inference.
• Hybrid PEFT combines ideas from multiple families to balance performance and efficiency.
• The survey also covers system-level considerations, including distributed tuning, PEFT query serving, and concurrent tuning, highlighting practical deployment costs and constraints.
• Applications span LLMs, Vision Transformers, vision-language models, and diffusion models, illustrating PEFT’s versatility across modalities.