[Paper Review] Joint Training of Low-Precision Neural Network with Quantization Interval Parameters

Sangil Jung, Changyong Son+5 more|arXiv (Cornell University)|Aug 17, 2018

Advanced Neural Network Applications10 references18 citations

TL;DR

This paper proposes a trainable quantizer that learns optimal quantization intervals for low-precision neural networks, directly minimizing task loss to maintain high accuracy even at 4-bit, 3-bit, and 2-bit precision. The method enables effective quantization of pretrained models without access to training data, outperforming prior approaches on ImageNet across ResNet-18, -34, and AlexNet.

ABSTRACT

Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.

Motivation & Objective

• To address the significant accuracy degradation caused by low-bit quantization in deep neural networks.
• To develop a trainable quantizer that learns optimal quantization intervals during training.
• To enable quantization of pretrained models without requiring access to their original training data.
• To maintain high accuracy at ultra-low bit-widths (e.g., 2-bit, 3-bit) while minimizing performance drop.
• To outperform existing quantization methods on standard benchmarks like ImageNet.

Proposed method

• The method parameterizes quantization intervals as learnable parameters within the network.
• Quantization intervals are optimized end-to-end by backpropagating through the quantization function using differentiable relaxation.
• The quantizer transforms and discretizes both activations and weights using these learned intervals.
• The approach allows joint training of low-precision networks with quantization intervals updated via gradient descent.
• The method supports heterogeneous datasets, enabling transfer learning and quantization of pretrained models.
• The quantizer is trained using the full task loss, ensuring alignment with network objectives.

Experimental results

Research questions

• RQ1 Can quantization intervals be effectively learned during training to reduce accuracy degradation in low-precision networks?
• RQ2 How well does the proposed method maintain accuracy when reducing bit-widths to 2-bit and 3-bit?
• RQ3 Can the quantizer be applied to pretrained models without access to their training data?
• RQ4 Does the method outperform existing quantization techniques on standard benchmarks like ImageNet?
• RQ5 Can the method generalize across different network architectures such as ResNet and AlexNet?

Key findings

• The proposed method achieves state-of-the-art accuracy on ImageNet at 4-bit precision, matching the performance of full-precision 32-bit networks.
• The method significantly reduces accuracy degradation when further reducing bit-width to 3-bit and 2-bit compared to existing approaches.
• The model maintains high accuracy when quantizing pretrained networks without requiring access to their original training data.
• The method outperforms existing quantization techniques on ResNet-18, ResNet-34, and AlexNet architectures.
• The end-to-end differentiable training of quantization intervals leads to better convergence and performance than fixed or heuristic interval settings.