[Paper Review] INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation

Yuji Chai, John Gkountouras+3 more|arXiv (Cornell University)|Jun 13, 2023

Topic Modeling4 citations

TL;DR

This paper introduces INT2.1, a method that enables fine-tuning of INT2 quantized large language models on consumer hardware by combining Extremely Memory-Efficient Fine-tuning (EMEF) with Low-Rank Error Correction (LREC). It reduces VRAM usage by up to 5.6× and achieves coherent text generation from a fully functional INT2 LLM, marking the first such model to reach this performance level with only a 1.05× model size overhead.

ABSTRACT

We introduce a method that dramatically reduces fine-tuning VRAM requirements and rectifies quantization errors in quantized Large Language Models. First, we develop an extremely memory-efficient fine-tuning (EMEF) method for quantized models using Low-Rank Adaptation (LoRA), and drawing upon it, we construct an error-correcting algorithm designed to minimize errors induced by the quantization process. Our method reduces the memory requirements by up to 5.6 times, which enables fine-tuning a 7 billion parameter Large Language Model (LLM) on consumer laptops. At the same time, we propose a Low-Rank Error Correction (LREC) method that exploits the added LoRA layers to ameliorate the gap between the quantized model and its float point counterpart. Our error correction framework leads to a fully functional INT2 quantized LLM with the capacity to generate coherent English text. To the best of our knowledge, this is the first INT2 Large Language Model that has been able to reach such a performance. The overhead of our method is merely a 1.05 times increase in model size, which translates to an effective precision of INT2.1. Also, our method readily generalizes to other quantization standards, such as INT3, INT4, and INT8, restoring their lost performance, which marks a significant milestone in the field of model quantization. The strategies delineated in this paper hold promising implications for the future development and optimization of quantized models, marking a pivotal shift in the landscape of low-resource machine learning computations.

Motivation & Objective

• Address the high VRAM requirements of fine-tuning quantized large language models, especially at INT2 precision.
• Enable fine-tuning of 7B-parameter LLMs on consumer-grade laptops with limited VRAM (e.g., 6GB).
• Correct quantization-induced performance degradation in low-bit quantized models without relying on specific quantization schemes.
• Develop a generalizable, parameter-efficient error correction mechanism that leverages LoRA parameters to restore model performance.

Proposed method

• Propose an Extremely Memory-Efficient Fine-tuning (EMEF) framework that applies Low-Rank Adaptation (LoRA) to quantized models, drastically reducing VRAM usage during fine-tuning.
• Introduce Low-Rank Error Correction (LREC), a quantization-agnostic error correction mechanism that uses the floating-point LoRA parameters to correct quantization-induced errors.
• Integrate EMEF and LREC into a unified training pipeline that maintains model efficiency while restoring performance lost due to INT2 quantization.
• Use low-rank decomposition matrices (LoRA) as both fine-tuning parameters and error correction components, enabling dual-purpose parameter injection.
• Design the method to be compatible with various quantization standards, including INT3, INT4, and INT8, by remaining agnostic to the quantization method.
• Optimize inference and training kernels to reduce memory bandwidth bottlenecks, though current implementation is limited by inefficient GPU tensor core utilization.

Experimental results

Research questions

• RQ1 Can INT2 quantized LLMs be fine-tuned efficiently on consumer-grade hardware with limited VRAM?
• RQ2 To what extent can LoRA-based parameters correct performance degradation caused by extreme quantization (e.g., INT2)?
• RQ3 Does a quantization-agnostic error correction mechanism exist that improves performance across different quantization levels without retraining?
• RQ4 What is the trade-off between model size increase, quantization bitwidth, and downstream performance in low-bit fine-tuning?
• RQ5 Can a fully functional, coherent-text-generating LLM be achieved at INT2 precision using only a 1.05× model size overhead?

Key findings

• The EMEF framework reduces VRAM requirements for fine-tuning a 7B-parameter LLM by up to 5.6×, enabling training on consumer GPUs with as little as 6GB of VRAM.
• The LREC method effectively reduces perplexity in extremely low-bit scenarios, demonstrating significant performance recovery from quantization-induced errors.
• The proposed INT2.1 model achieves coherent English text generation, representing the first functional INT2 LLM to do so at scale.
• The method incurs only a 1.05× increase in model size, effectively achieving an effective precision of INT2.1 with minimal overhead.
• The framework generalizes across quantization standards, restoring performance for INT3, INT4, and INT8 models through the same error correction mechanism.
• Despite theoretical advantages in memory bandwidth, current implementations are slower than FP16 baselines due to inefficient GPU tensor core utilization.