[Paper Review] Learning Low Precision Deep Neural Networks through Regularization.

Yoojin Choi, Mostafa El‐Khamy+1 more|arXiv (Cornell University)|Sep 1, 2018

Advanced Image Processing Techniques64 references27 citations

TL;DR

This paper proposes a regularization-based method to train low-precision deep neural networks (DNNs) with minimal quantization loss by introducing a learnable regularization coefficient. The approach enables state-of-the-art accuracy for 8-bit and binary DNNs on ImageNet benchmarks, achieving 0.5 dB PSNR loss in super-resolution tasks and up to 401× compression in LeNet-5 through pruning.

ABSTRACT

We consider the quantization of deep neural networks (DNNs) to produce low-precision models for efficient inference of fixed-point operations. Compared to previous approaches to training quantized DNNs directly under the constraints of low-precision weights and activations, we learn the quantization of DNNs with minimal quantization loss through regularization. In particular, we introduce the learnable regularization coefficient to find accurate low-precision models efficiently in training. In our experiments, the proposed scheme yields the state-of-the-art low-precision models of AlexNet and ResNet-18, which have better accuracy than their previously available low-precision models. We also examine our quantization method to produce low-precision DNNs for image super resolution. We observe only $0.5$~dB peak signal-to-noise ratio (PSNR) loss when using binary weights and 8-bit activations. The proposed scheme can be used to train low-precision models from scratch or to fine-tune a well-trained high-precision model to converge to a low-precision model. Finally, we discuss how a similar regularization method can be adopted in DNN weight pruning and compression, and show that $401 imes$ compression is achieved for LeNet-5.

Motivation & Objective

• To reduce quantization loss in low-precision DNNs without relying on end-to-end training under strict precision constraints.
• To enable efficient training of low-precision models from scratch or via fine-tuning of high-precision models.
• To explore the applicability of the regularization method beyond quantization, including weight pruning and model compression.
• To achieve high-accuracy low-precision models for image super-resolution and classification tasks.

Proposed method

• Introduce a learnable regularization coefficient that dynamically adjusts during training to minimize quantization error.
• Apply regularization to the weight and activation quantization process, allowing the network to learn optimal low-precision representations.
• Train models end-to-end with quantization-aware regularization, enabling convergence to accurate low-precision solutions.
• Use the same regularization framework for both weight quantization and activation quantization in a unified training procedure.
• Adapt the method to weight pruning by applying the same regularization principle, enabling high compression ratios.
• Apply the method to both training from scratch and fine-tuning pre-trained high-precision models.

Experimental results

Research questions

• RQ1 Can regularization with a learnable coefficient reduce quantization loss more effectively than direct quantization training?
• RQ2 How does the proposed method compare to existing quantization schemes in terms of accuracy on ImageNet?
• RQ3 To what extent can the method preserve performance in low-precision settings for image super-resolution tasks?
• RQ4 Can the same regularization framework be extended to model compression and pruning?
• RQ5 What compression ratios can be achieved using this regularization-based pruning approach?

Key findings

• The proposed method achieves state-of-the-art accuracy for low-precision models of AlexNet and ResNet-18.
• For image super-resolution, the method incurs only 0.5 dB PSNR loss when using binary weights and 8-bit activations.
• The method enables training low-precision models from scratch or fine-tuning high-precision models with consistent performance gains.
• The regularization framework achieves up to 401× model compression for LeNet-5 through weight pruning.
• The learnable regularization coefficient effectively balances quantization error and model accuracy during training.