[Paper Review] Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks

Ruihao Gong, Xianglong Liu+6 more|arXiv (Cornell University)|Aug 14, 2019

Advanced Neural Network Applications37 references60 citations

TL;DR

DSQ introduces a differentiable, evolving soft quantization that bridges full-precision and low-bit networks, improving training stability and accuracy for 1–4 bit quantization and enabling faster ARM deployment.

ABSTRACT

Hardware-friendly network quantization (e.g., binary/uniform quantization) can efficiently accelerate the inference and meanwhile reduce memory consumption of the deep neural networks, which is crucial for model deployment on resource-limited devices like mobile phones. However, due to the discreteness of low-bit quantization, existing quantization methods often face the unstable training process and severe performance degradation. To address this problem, in this paper we propose Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks. DSQ can automatically evolve during training to gradually approximate the standard quantization. Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation, and reduce the quantization loss in forward process with an appropriate clipping range. Extensive experiments over several popular network structures show that training low-bit neural networks with DSQ can consistently outperform state-of-the-art quantization methods. Besides, our first efficient implementation for deploying 2 to 4-bit DSQ on devices with ARM architecture achieves up to 1.7$ imes$ speed up, compared with the open-source 8-bit high-performance inference framework NCNN. [31]

Motivation & Objective

• Motivate the challenge of training low-bit networks due to discreteness and gradient mismatch.
• Propose a differentiable quantization function that gradually approximates standard quantization during training.
• Balance clipping and rounding errors to reduce quantization loss.
• Enable hardware-friendly deployment with efficient 2–4 bit kernels.
• Provide a flexible framework compatible with binary and uniform quantization.

Proposed method

• Define a differentiable asymptotic quantizer using piecewise tanh-based functions to approximate standard quantization.
• Introduce a characteristic variable alpha to control DSQ's approximation quality and an evolution training strategy to optimize alpha during training.
• Jointly optimize clipping bounds (l and u) with alpha to balance clipping and rounding errors.
• Provide backpropagation equations and an algorithm for training DSQ-enabled networks.
• Demonstrate hardware-efficient 2–4 bit GEMM kernels on ARM NEON for deployment.

Experimental results

Research questions

• RQ1 How can quantization be made differentiable to provide accurate gradients during training?
• RQ2 Can an evolving quantization function bridge full-precision and low-bit networks without destabilizing training?
• RQ3 How should clipping bounds and the approximation strength be managed to balance clipping and rounding errors?
• RQ4 What performance gains can be achieved with DSQ on CIFAR-10 and ImageNet across common architectures?
• RQ5 Is DSQ compatible with both binary and uniform quantization and with existing quantization methods?

Key findings

• DSQ consistently improves accuracy over state-of-the-art quantization methods across CIFAR-10 and ImageNet.
• The evolution of alpha helps DSQ gradually approximate standard quantization, improving convergence and stability.
• Joint optimization of clipping values and alpha balances clipping and rounding errors, reducing quantization loss.
• DSQ enhances 1/1 bit quantization performance and complements methods like PACT, achieving higher top-1/top-5 accuracy.
• Efficient 2–4 bit ARM NEON kernels enable faster inference, surpassing existing open-source frameworks on ARM hardware.