[Paper Review] QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni+2 more|arXiv (Cornell University)|May 23, 2023

Ferroelectric and Negative Capacitance Devices485 citations

TL;DR

QLoRA finetunes quantized 4-bit LMs using Low Rank Adapters to match 16-bit performance with far less memory, enabling large models to be trained on a single GPU.

ABSTRACT

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

Motivation & Objective

• Motivate memory-efficient finetuning of very large language models (LLMs).
• Propose 4-bit NF4 quantization combined with Low Rank Adapters to preserve performance.
• Demonstrate scalability across 7B–65B models and multiple instruction-following benchmarks.
• Analyze data quality vs. dataset size and evaluate chatbot performance using human and GPT-4 evaluations.

Proposed method

• Quantize pretrained weights to 4-bit NF4 with Double Quantization to reduce memory footprint.
• Backpropagate gradients through quantized weights to learn LoRA adapters inserted at every transformer layer.
• Use Paged Optimizers and NVIDIA unified memory to handle memory spikes during training.
• Maintain a computation path that dequantizes to BF16 for forward and backward passes while updating only LoRA parameters.
• Evaluate against full 16-bit finetuning and 16-bit LoRA baselines across multiple model sizes and datasets.

Figure 1: Different finetuning methods and their memory requirements. QLoRA improves over LoRA by quantizing the transformer model to 4-bit precision and using paged optimizers to handle memory spikes.

Experimental results

Research questions

• RQ1 Can 4-bit NF4 quantization with Double Quantization retain full 16-bit finetuning performance when paired with LoRA adapters?
• RQ2 How does QLoRA scale across model sizes (7B–65B) and instruction datasets in terms of accuracy and memory efficiency?
• RQ3 Do data quality and dataset choice trump dataset size for instruction tuning and chatbot capabilities?
• RQ4 How do GPT-4 and human evaluations compare for ranking chatbots trained with QLoRA on Vicuna/OASST1 benchmarks?

Key findings

• QLoRA with NF4 + Double Quantization matches 16-bit full finetuning and 16-bit LoRA performance on academic benchmarks.
• NF4 outperforms FP4 and Int4 in zero-shot and finetuning tasks, with Double Quantization reducing memory without degrading performance.
• Guanaco models (7B–65B) achieve near ChatGPT performance on Vicuna benchmarks, with Guanaco 65B reaching 99.3% of ChatGPT in GPT-4 evaluated Vicuna results.
• 4-bit QLoRA enables finetuning of 65B models on a single 48GB GPU, and 33B on 21GB, with substantial memory savings vs. full 16-bit finetuning.
• Datasets’ quality matters more than size for instruction following and chatbot performance, and strong MMLU performance does not always imply superior chatbot performance.
• GPT-4 based evaluations largely align with human judgments in tournament-style chatbot comparisons, but there are notable disagreements that warrant careful interpretation.

Figure 5: The crowdsourcing form used by human annotators.