[Paper Review] LoRA: Low-Rank Adaptation of Large Language Models

J. Edward Hu, Yelong Shen+6 more|arXiv (Cornell University)|Jun 17, 2021

Topic Modeling57 references2,389 citations

TL;DR

LoRA freezes pre-trained weights and injects trainable low-rank matrices to adapt transformers, achieving comparable or better performance with far fewer trainable parameters and no extra inference latency.

ABSTRACT

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

Motivation & Objective

• Motivate efficient adaptation of large pre-trained language models to downstream tasks without full fine-tuning.
• Propose a low-rank update mechanism that preserves frozen base weights while enabling task-specific learning.
• Demonstrate that low-rankAdaptation can match or surpass full fine-tuning with far fewer trainable parameters and reduced resource usage.
• Provide empirical evidence across RoBERTa, DeBERTa, GPT-2, and GPT-3 (GPT-3 175B) and offer practical tooling for integration.

Proposed method

• Represent weight updates as a low-rank decomposition: ΔW = BA with W0 + ΔW, where W0 is frozen, B ∈ R^{d×r}, A ∈ R^{r×k} and r ≪ min(d,k).
• Train only A and B while keeping W0 fixed, effectively reducing the number of trainable parameters.
• Scale the update by α/r to stabilize optimization when varying rank r.
• Apply LoRA primarily to self-attention projections (Wq, Wk, Wv, Wo) in Transformer layers while freezing MLP modules for efficiency.
• During inference, merge W0 and BA to avoid additional latency, enabling rapid task switching by swapping LoRA weights.
• LoRA parameters scale with rank r and number of adapted matrices, enabling substantial parameter efficiency (e.g., very small r yields tiny Θ).
• Demonstrate compatibility with other adaptation methods and provide PyTorch tooling and checkpoints for RoBERTa, DeBERTa, and GPT-2.

Experimental results

Research questions

• RQ1 What subset of Transformer weight matrices should LoRA adapt for optimal downstream performance under a fixed parameter budget?
• RQ2 Is the learned weight update ΔW inherently low rank, and what rank r is effective in practice?
• RQ3 How does ΔW relate to the frozen weights W0, and to what extent does ΔW correlate with W0?
• RQ4 Can LoRA scale to very large models (e.g., GPT-3 175B) while maintaining or improving task performance compared to full fine-tuning?

Key findings

• LoRA can reduce trainable parameters by up to roughly 10,000x compared with full fine-tuning on GPT-3 175B.
• VRAM usage during training can be reduced by up to about 2/3 since gradients for the frozen parameters need not be stored.
• Training throughput with LoRA can exceed full fine-tuning (e.g., 43.1 tokens/s per V100 vs 32.5 tokens/s per V100 for GPT-3 175B in comparable setups).
• LoRA matches or exceeds fine-tuning performance on RoBERTa, DeBERTa, GPT-2, and GPT-3 across multiple tasks, despite using far fewer trainable parameters.
• On GPT-3 175B, LoRA with small ranks (e.g., r as low as 2–8 per adapted matrix) yields strong performance and favorable scalability across tasks like WikiSQL and MNLI-matched.
• LoRA enables no additional inference latency, since the adapted weights can be merged with the frozen weights for deployment.