[Paper Review] SliderQuant: Accurate Post-Training Quantization for LLMs

Shigeng Wang, Chao Li+4 more|arXiv (Cornell University)|Mar 26, 2026

Multimodal Machine Learning Applications0 citations

TL;DR

SliderQuant introduces an adaptive sliding quantization framework that differentially quantizes LLM layers (shallow, intermediate, deep) to improve post-training quantization at low bit-widths, outperforming prior PTQ methods on weight-only and weight-activation quantization across diverse models and tasks.

ABSTRACT

In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.

Motivation & Objective

• Motivate the need for layer-aware PTQ in LLMs under aggressive quantization (e.g., 4-bit).
• Empirically characterize layer sensitivity to quantization across shallow, intermediate, and deep layers.
• Propose SliderQuant with inter-layer and intra-layer sliding components to reduce cross-layer quantization error.
• Demonstrate broad effectiveness across model families (Llama, Qwen), sizes, and tasks (generation, commonsense, math/code).
• Show compatibility with weight-only and weight-activation PTQ, including variants with rotation transformations.

Proposed method

• Introduce adaptive sliding quantization as a generalization of fixed-size sliding PTQ.
• Develop inter-layer sliding quantization with three window designs: progressively expanded (shallow), fixed-size (intermediate), progressively contracted (deep).
• Intra-layer sliding quantization incrementally quantizes all layers within a window using progressive expansion inside the window.
• Use learnable parameters (channel-wise scale alpha, low-rank refinements A,B) plus a uniform quantizer to minimize mean square error between F(W,X) and F(What,X).
• Combine CS (channel scaling) and LoRA-inspired refinements to obtain refined W and X before quantization (Eq. 2).
• Provide SliderQuant and a costlier variant SliderQuant+ with rotation-based inference-time enhancements.

Experimental results

Research questions

• RQ1 Do different layers in modern LLMs exhibit distinct sensitivity to quantization that warrants layer-aware PTQ design?
• RQ2 Can adaptive sliding windows reduce cross-layer quantization error more effectively than fixed-size sliding or layer-wise methods?
• RQ3 How does SliderQuant perform for weight-only and weight-activation quantization across diverse models and tasks?
• RQ4 Is SliderQuant effective on complex architectures (MoE) and distilled models with chain-of-thought?

Key findings

• SliderQuant consistently reduces perplexity (WikiText2, C4) compared with RTN, GPTQ, OmniQuant, CBQ, and others across multiple models and bit-widths (including W4A4).
• In commonsense QA benchmarks, SliderQuant improves average accuracy over other PTQ methods (e.g., on Qwen2.5-14B and Llama2-13B).
• SliderQuant+ with rotation transformations achieves the best results among methods with extra inference-time costs on several models and benchmarks.
• The method generalizes to MoE architectures (Qwen3-30B-A3B) and to DeepSeek-R1 distilled models, maintaining strong performance under low-bit quantization.
• Across tasks, SliderQuant shows robust gains under both weight-only and weight-activation quantization, often outperforming rotation-augmented baselines.
• The framework remains competitive with or without extra inference-time costs, demonstrating broad applicability and scalability to large LLMs.