[论文解读] LoRA: Low-Rank Adaptation of Large Language Models

J. Edward Hu, Yelong Shen+6 more|arXiv (Cornell University)|Jun 17, 2021

Topic Modeling参考文献 57被引用 2,389

一句话总结

LoRA 冻结预训练权重并注入可训练的低秩矩阵以适配变换器，从而在显著更少的可训练参数且没有额外推理延迟的情况下实现相媲美或更优的性能。

ABSTRACT

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

研究动机与目标

• 在不进行完全微调的情况下，推动大规模预训练语言模型对下游任务的高效适应。
• 提出一种保持冻结的基础权重同时实现任务特定学习的低秩更新机制。
• 证明 low-rankAdaptation 在可训练参数显著更少、资源使用降低的情况下，能够达到甚至超过全微调的性能。
• 提供跨 RoBERTa、DeBERTa、GPT-2 以及 GPT-3（GPT-3 175B）的实证证据，并提供可用于集成的实用工具。

提出的方法

• 将权重更新表示为低秩分解：ΔW = BA，其中 W0 + ΔW，W0 为冻结，B ∈ R^{d×r}，A ∈ R^{r×k}，且 r ≪ min(d,k)。
• 仅训练 A 和 B，同时保持 W0 不变，有效减少可训练参数数量。
• 在变动秩 r 时，通过 α/r 缩放更新以稳定优化。
• 将 LoRA 主要应用于 Transformer 层中的自注意力投影（Wq、Wk、Wv、Wo），同时冻结 MLP 模块以提高效率。
• 在推理阶段，将 W0 与 BA 合并以避免额外延迟，通过切换 LoRA 权重实现快速任务切换。
• LoRA 参数与秩 r 以及被适应矩阵的数量成比例增长，从而实现显著的参数效率（例如，r 很小时 Θ 也非常小）。
• 展示与其他适配方法的兼容性，并提供用于 RoBERTa、DeBERTa 和 GPT-2 的 PyTorch 工具和检查点。

实验结果

研究问题

• RQ1 在固定参数预算下，LoRA 应适应 Transformer 权重矩阵的哪一子集以实现最佳下游性能？
• RQ2 Is the learned weight update ΔW inherently low rank, and what rank r is effective in practice?
• RQ3 ΔW 与冻结权重 W0 之间有何关系，ΔW 与 W0 的相关程度到何种程度？
• RQ4 Can LoRA scale to very large models (e.g., GPT-3 175B) while maintaining or improving task performance compared to full fine-tuning?

主要发现

• LoRA can reduce trainable parameters by up to roughly 10,000x compared with full fine-tuning on GPT-3 175B.
• VRAM usage during training can be reduced by up to about 2/3 since gradients for the frozen parameters need not be stored.
• Training throughput with LoRA can exceed full fine-tuning (e.g., 43.1 tokens/s per V100 vs 32.5 tokens/s per V100 for GPT-3 175B in comparable setups).
• LoRA matches or exceeds fine-tuning performance on RoBERTa, DeBERTa, GPT-2, and GPT-3 across multiple tasks, despite using far fewer trainable parameters.
• On GPT-3 175B, LoRA with small ranks (e.g., r as low as 2–8 per adapted matrix) yields strong performance and favorable scalability across tasks like WikiSQL and MNLI-matched.
• LoRA enables no additional inference latency, since the adapted weights can be merged with the frozen weights for deployment.