[Paper Review] Parameter-Efficient Finetuning of Transformers for Source Code

Shamil Ayupov, Nadezhda Chirkova|arXiv (Cornell University)|Dec 12, 2022

Software Engineering Research4 citations

TL;DR

This paper evaluates parameter-efficient fine-tuning (PEFT) methods—LoRA and adapters—on pretrained code models (CodeT5 and PLBART) across four code-processing tasks. It finds that while PEFT matches or exceeds full fine-tuning in code understanding tasks, it underperforms significantly in code generation, highlighting domain-specific limitations of PEFT in source code NLP.

ABSTRACT

Pretrained Transformers achieve state-of-the-art performance in various code-processing tasks but may be too large to be deployed. As software development tools often incorporate modules for various purposes which may potentially use a single instance of the pretrained model, it appears relevant to utilize parameter-efficient fine-tuning for the pretrained models of code. In this work, we test two widely used approaches, adapters and LoRA, which were initially tested on NLP tasks, on four code-processing tasks. We find that though the efficient fine-tuning approaches may achieve comparable or higher performance than the standard, full, fine-tuning in code understanding tasks, they underperform full fine-tuning in code-generative tasks. These results underline the importance of testing efficient fine-tuning approaches on other domains than NLP and motivate future research in efficient fine-tuning for source code.

Motivation & Objective

• To assess the effectiveness of parameter-efficient fine-tuning (PEFT) methods—LoRA and adapters—on source code processing tasks.
• To compare PEFT approaches (LoRA, adapters, and their combination) against full fine-tuning in terms of performance and parameter efficiency.
• To evaluate whether PEFT methods, effective in NLP, generalize well to code-specific tasks with potentially noisier and more complex data.
• To investigate the impact of model size and hyperparameters (e.g., rank r) on PEFT performance in code generation and code understanding.
• To provide empirical evidence on the suitability of PEFT for deployment in software development tools like IDEs, where model efficiency is critical.

Proposed method

• Applies LoRA (Low-Rank Adaptation) by introducing low-rank matrices $\Delta W = AB$ to update original model weights $W$, with $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, and $r < \min(d,k)$, applied to attention and feed-forward layers.
• Implements Adapter Training (AT) by inserting small feed-forward networks after attention and feed-forward blocks in the Transformer, with only the adapter parameters updated during fine-tuning.
• Combines LoRA and adapters (FF-LoRA + AT) to explore synergistic improvements while monitoring parameter count and inference overhead.
• Uses CodeT5-base (223M parameters) and PLBART-base (140M parameters) as base models, fine-tuned on four tasks from the CodeXGLEU benchmark: code summarization, code clone detection, code translation, and code generation.
• Employs HuggingFace’s Transformers library and evaluates performance using standard metrics: BLEU-4, EM, CodeBLEU, and exact match.
• Tunes hyperparameters (e.g., rank $r$) on validation sets and reports results averaged over three runs with standard deviation.

Experimental results

Research questions

• RQ1 How do LoRA and adapter-based PEFT methods compare to full fine-tuning in code understanding tasks like code summarization and clone detection?
• RQ2 To what extent do PEFT methods maintain performance in code generation tasks such as code translation and code-to-text generation?
• RQ3 Does combining LoRA and adapters improve performance while keeping parameter efficiency?
• RQ4 How does the choice of rank $r$ affect the performance and parameter count of PEFT methods in code-specific tasks?
• RQ5 Can PEFT methods effectively handle noisy, real-world code data collected from open-source repositories, especially in low-data regimes?

Key findings

• In code summarization (Java), FF-LoRA with $r=8$ achieved a BLEU-4 score of 18.10, outperforming full fine-tuning (17.32) on the same task.
• For code translation (C# → Java), the combination of FF-LoRA and AT with $r=16$ achieved a BLEU-4 score of 76.89, surpassing full fine-tuning (79.14) in a relative sense but still below the baseline in absolute terms.
• In code generation tasks, LoRA and adapters consistently underperformed full fine-tuning: LoRA achieved 74.26 BLEU-4 on code translation (vs. 79.14 for full fine-tuning), indicating a significant performance gap.
• Adapter Training with $r=16$ achieved 75.30 BLEU-4 on code translation, still below the full fine-tuning baseline of 78.60, suggesting limited capacity for complex generation.
• The combination of FF-LoRA and AT achieved the highest BLEU-4 (76.89) and CodeBLEU (82.53) on code translation, but still fell short of full fine-tuning’s 79.14 BLEU-4 and 84.33 CodeBLEU.
• Qualitative analysis showed that PEFT methods often produced shorter or less precise summaries than full fine-tuning, with AT models sometimes introducing factual errors (e.g., "Detects the operating system" vs. "Detect operating system").