[論文レビュー] Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Charith Chandra Sai Balne, Sreyoshi Bhaduri+3 more|arXiv (Cornell University)|Apr 21, 2024

Neural Networks and Applications被引用数 5

ひとこと要約

この調査はドメインを横断してParameter Efficient Fine-Tuning (PEFT) メソッドを分析し、LoRA、DoRA、LoReFT などの戦略を比較し、それらの適用とトレードオフを概観します。

ABSTRACT

The rise of deep learning has marked significant progress in fields such as computer vision, natural language processing, and medical imaging, primarily through the adaptation of pre-trained models for specific tasks. Traditional fine-tuning methods, involving adjustments to all parameters, face challenges due to high computational and memory demands. This has led to the development of Parameter Efficient Fine-Tuning (PEFT) techniques, which selectively update parameters to balance computational efficiency with performance. This review examines PEFT approaches, offering a detailed comparison of various strategies highlighting applications across different domains, including text generation, medical imaging, protein modeling, and speech synthesis. By assessing the effectiveness of PEFT methods in reducing computational load, speeding up training, and lowering memory usage, this paper contributes to making deep learning more accessible and adaptable, facilitating its wider application and encouraging innovation in model optimization. Ultimately, the paper aims to contribute towards insights into PEFT's evolving landscape, guiding researchers and practitioners in overcoming the limitations of conventional fine-tuning approaches.

研究の動機と目的

• Provide a comprehensive overview of recent advances in PEFT methods.
• Compare PEFT strategies in terms of efficiency, training speed, and memory usage.
• Highlight applications across text, vision, biology, and speech domains.
• Identify challenges and future research directions to democratize deep learning via PEFT.

提案手法

• Discuss foundational issues with full fine-tuning and the motivation for PEFT.
• Present a representative PEFT equation: DII(b,s,R)=b+R^{ ext{top}}(Rs-Rb) to guide hidden-state alignment.
• Compare a range of PEFT techniques (LoRA, DoRA, Prefix Tuning, BitFit, etc.) and their parameter reductions.
• Summarize performance and efficiency trade-offs across datasets and model sizes.
• Illustrate empirical findings through cross-domain applications and tables of results.

Figure 1: Comparative study of PEFT across different applications.

実験結果

リサーチクエスチョン

• RQ1 How do PEFT methods compare in terms of parameter efficiency and performance across various tasks?
• RQ2 What are the key trade-offs and limitations of PEFT approaches in different domains (NLP, vision, biology, audio)?
• RQ3 Which PEFT techniques offer best practical balance between accuracy, training speed, and resource usage across applications?

主な発見

Method	Parameter reduction (%)	Advantages	Disadvantages
フルファインチューニング (ViT-B/16, BARD)	0	Performant baseline	High memory footprint (33B parameters)
アダプター・モジュール（Tiny）	85	Flexible, modular design	Requires hyperparameter tuning
アダプター・モジュール（Small）	75	Flexible, modular design	Requires hyperparameter tuning
LoRA	90	Memory efficient (3.3B parameters)	Limited control over updates
LoReFT	70-90	Memory efficient, potentially interpretable	Efficiency depends on task and hyperparameters
Prefix Tuning（Learned）	65	Simple implementation	May not capture complex video features
Sparse Fine-Tuning (40% pruning)	60	Memory efficient (13.2B parameters)	Requires careful selection of parameters
Sparse Fine-Tuning (80% pruning)	80	Extremely memory efficient (6.6B parameters)	Significant accuracy drop at high pruning ratio
BitFit (8-bit)	95	Extremely memory efficient (1.65B parameters)	Limited performance gains in high-data regime

• LoRA generally achieves strong parameter efficiency with notable performance gains.
• LoReFT can outperform several PEFT methods on commonsense reasoning with very small trainable parameter fractions.
• In arithmetic reasoning, LoRA and adapters often outperform LoReFT, indicating task-dependent effectiveness.
• Across applications, PEFT methods significantly reduce training resources while maintaining competitive performance.
• 3D and video-text tasks show notable gains with specialized adapters like AGAdapter and KAdaptation, confirming cross-domain effectiveness.

Figure 2: Illustration of workflow for the PEFT paradigm starting with a pre-trained model ( $\theta$ ), to which modifications such as additions, specifications, and reparameterizations are applied, effectively differentiating between frozen and tunable parameters to enhance model performance.