[Paper Review] LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression

Ayush Kaushal, Tejas Vaidhya+1 more|arXiv (Cornell University)|Sep 25, 2023

Computational Physics and Python Applications4 citations

TL;DR

This paper proposes LoRD, a one-shot low-rank decomposition method for compressing monolingual code LLMs without retraining. By factorizing dense linear layers into lower-rank matrices, LoRD achieves up to 39.58% rank reduction with less than 1% perplexity increase, enabling 22.35% faster inference and compressing StarCoder 16B to 13.2B parameters with no performance drop.

ABSTRACT

Low Rank Decomposition of matrix - splitting a large matrix into a product of two smaller matrix offers a means for compression that reduces the parameters of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain fully differentiable and all the parameters trainable, while being able to leverage the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via Low Rank Decomposition (LoRD) and observe that ranks for the linear layers in these models can be reduced by upto 39.58% with less than 1% increase in perplexity. We then use Low Rank Decomposition (LoRD) to compress StarCoder 16B to 13.2B parameter with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less than 10 minutes on a single A100. The compressed models speeds up inference by up to 22.35% with just a single line of change in code over huggingface's implementation with pytorch backend. Low Rank Decomposition (LoRD) models remain compatible with state of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over Low Rank Decomposition (LoRD) model further reduces memory requirements by as much as 21.2% over vanilla QLoRA while offering similar gains from parameter efficient fine tuning. Our work shows Low Rank Decomposition (LoRD) as a promising new paradigm for LLM compression.

Motivation & Objective

• To explore low-rank decomposition as a viable, retraining-free compression method for large monolingual code LLMs.
• To address the limitations of quantization and pruning—such as hardware-specific kernels and non-differentiability—by leveraging dense matrix multiplication.
• To enable fast, hardware-optimized inference through efficient, fully differentiable, and trainable compressed models.
• To demonstrate compatibility with state-of-the-art quantization and parameter-efficient fine-tuning methods like QLoRA.
• To enable deployment of large code LLMs on consumer-grade hardware by reducing memory and compute demands.

Proposed method

• LoRD decomposes each dense linear layer in a transformer block into two smaller low-rank matrices, reducing parameter count via matrix factorization.
• The method applies singular value decomposition (SVD) or data-aware decomposition to identify low-rank approximations of weight matrices without retraining.
• Compression is performed in a single shot using the full model weights, avoiding iterative fine-tuning or distillation.
• The resulting decomposed layers are fully differentiable and compatible with existing floating-point GEMM kernels (e.g., cuBLAS), enabling high-speed inference.
• The approach integrates seamlessly with Hugging Face’s PyTorch backend, requiring only one line of code change for deployment.
• LoRD models are further compressed using near-lossless quantization (SpQR) and QLoRA fine-tuning, enabling additional gains in model size and memory efficiency.

(a) Perplexity vs % Rank Reduction for CodeGen Models.

Experimental results

Research questions

• RQ1 Can low-rank decomposition be effectively applied to large monolingual code LLMs (e.g., StarCoder, CodeGen) without retraining?
• RQ2 To what extent can the rank of linear layers in code LLMs be reduced while maintaining model performance?
• RQ3 How does LoRD compare to quantization and pruning in terms of inference speed, hardware compatibility, and model differentiability?
• RQ4 Can LoRD be combined with existing quantization and parameter-efficient fine-tuning methods to achieve further compression and efficiency gains?
• RQ5 Does the decomposed model maintain performance when deployed on consumer-grade GPUs with limited memory?

Key findings

• LoRD enables up to 39.58% reduction in rank of linear layers in code LLMs with less than 1% increase in perplexity.
• StarCoder 16B was compressed to 13.2B parameters with no drop in HumanEval Pass@1 score (31.67 vs. 31.57), and to 12.3B with only a minor drop (29.22).
• Inference speedup of up to 22.35% was achieved on a single A100 with just one line of code change over Hugging Face’s default implementation.
• LoRD models remain fully differentiable and compatible with state-of-the-art quantization (SpQR), enabling further compression gains.
• QLoRA fine-tuning over LoRD models reduced memory usage by up to 21.2% compared to vanilla QLoRA, while maintaining similar fine-tuning performance.
• The method is hardware-optimized, leveraging existing cuBLAS kernels and avoiding sparse matrix overhead, making it faster than sparse or quantized inference on most GPUs.

(b) Perplexity vs % Rank Reduction for StarCoder Models.