[Paper Review] LoRA Learns Less and Forgets Less

Dan Biderman, Jacob Portes+10 more|arXiv (Cornell University)|May 15, 2024

Robotics and Automated Systems16 citations

TL;DR

LoRA underperforms full finetuning on code and math tasks but provides stronger regularization, reducing forgetting and preserving diversity in generations.

ABSTRACT

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

Motivation & Objective

• Assess whether LoRA can match full finetuning performance on challenging code and math domains.
• Evaluate LoRA under two training regimes: instruction finetuning (IFT) and continued pretraining (CPT).
• Analyze forgetting of source-domain abilities when finetuning on target domains.
• Characterize the rank and module sensitivity of LoRA perturbations relative to full finetuning.
• Provide best-practice recommendations for configuring LoRA in practice.

Proposed method

• Compare LoRA and full finetuning on Llama-2 7B (and 13B in places) across code and math domains.
• Use two data regimes: instruction finetuning (~100K prompt-response pairs) and continued pretraining (~10B tokens).
• Evaluate target-domain performance with HumanEval (code) and GSM8K (math).
• Evaluate source-domain forgetting with HellaSwag, ARC-Challenge, and Winograd problems.
• Perform singular value decomposition to analyze perturbation rank of finetuning.
• Conduct hyperparameter sweeps over learning rates, target modules, and LoRA rank.

Experimental results

Research questions

• RQ1 Under what conditions does LoRA approximate full finetuning accuracy on code and math tasks?
• RQ2 Does LoRA mitigate forgetting of the source domain compared with full finetuning?
• RQ3 How do LoRA perturbation rank and target modules influence performance and regularization?
• RQ4 What hyperparameters most strongly affect LoRA performance and stability?
• RQ5 What practical guidelines maximize LoRA's effectiveness for instruction finetuning and continued pretraining?

Key findings

• LoRA underperforms full finetuning in code and math tasks across CPT and IFT, with larger gaps in code than math.
• LoRA forgets less of the source domain than full finetuning, indicating a regularization effect.
• LoRA provides stronger regularization than weight decay or attention dropout and maintains more diverse output generations.
• Full finetuning tends to produce high-rank perturbations, with perturbation ranks 10–100x higher than typical LoRA configurations.
• LoRA is highly sensitive to learning rate and to the choice of target modules, with module choice outweighing rank in impact.
• Best practice guidance favors using LoRA for instruction finetuning, targeting all modules with a modest rank, and training for several epochs.