[Paper Review] Quantization of Deep Neural Networks for Accurate Edge Computing

Wentao Chen, Hailong Qiu+8 more|arXiv (Cornell University)|Apr 25, 2021

Advanced Neural Network Applications37 references4 citations

TL;DR

This paper proposes incremental quantization as a regularization technique that can improve deep neural network accuracy on edge devices, demonstrating up to 4.23% accuracy gains in biomedical image segmentation, image classification, and speech recognition—while achieving 3.5x–6.4x memory reduction. The method leverages progressive weight quantization to reduce overfitting in large models, challenging the conventional belief that quantization only degrades performance.

ABSTRACT

Deep neural networks (DNNs) have demonstrated their great potential in recent years, exceeding the per-formance of human experts in a wide range of applications. Due to their large sizes, however, compressiontechniques such as weight quantization and pruning are usually applied before they can be accommodated onthe edge. It is generally believed that quantization leads to performance degradation, and plenty of existingworks have explored quantization strategies aiming at minimum accuracy loss. In this paper, we argue thatquantization, which essentially imposes regularization on weight representations, can sometimes help toimprove accuracy. We conduct comprehensive experiments on three widely used applications: fully con-nected network (FCN) for biomedical image segmentation, convolutional neural network (CNN) for imageclassification on ImageNet, and recurrent neural network (RNN) for automatic speech recognition, and experi-mental results show that quantization can improve the accuracy by 1%, 1.95%, 4.23% on the three applicationsrespectively with 3.5x-6.4x memory reduction.

Motivation & Objective

• To investigate whether quantization, traditionally seen as a source of accuracy degradation, can instead improve model performance through regularization.
• To evaluate the impact of incremental quantization on three real-world applications: biomedical image segmentation, ImageNet classification, and automatic speech recognition.
• To demonstrate that quantization can reduce overfitting in large models, leading to improved generalization and accuracy.
• To provide empirical evidence that memory reduction via quantization does not necessarily come at the cost of accuracy, and can sometimes enhance it.
• To explore optimal quantization bit-widths and parallel model configurations that maximize accuracy under memory constraints.

Proposed method

• The authors apply incremental quantization by progressively reducing the bit-width of network weights from full-precision to lower bit representations (e.g., 8-bit, 6-bit, 4-bit) in a controlled, step-by-step manner.
• They re-implement and evaluate state-of-the-art quantization methods under identical hardware and framework conditions to ensure fair comparison.
• The method includes training with quantized weights from the start, using techniques like dynamic fixed-point representation and weight scaling to preserve representational capacity.
• For each application, they test various bit-widths and parallel configurations (e.g., two parallel models) to identify optimal trade-offs between memory, computation, and accuracy.
• They analyze the effect of quantization on overfitting by comparing performance on large vs. small models, observing that overfitting is reduced in large models but exacerbated in small ones.
• The approach treats quantization as a form of regularization, leveraging its inherent weight constraint to improve generalization in overparameterized networks.

Experimental results

Research questions

• RQ1 Can quantization improve deep neural network accuracy instead of degrading it, particularly in overparameterized models?
• RQ2 What is the optimal bit-width for quantization that maximizes accuracy while minimizing memory usage across diverse DNN applications?
• RQ3 How does incremental quantization affect overfitting in large versus small neural networks?
• RQ4 Can parallel model configurations with quantized weights further enhance performance beyond single-model quantization?
• RQ5 Does quantization-induced regularization lead to consistent accuracy improvements across different DNN architectures and tasks?

Key findings

• Incremental quantization improved accuracy by 1% on the MICCIA Gland dataset for biomedical image segmentation, outperforming the current state-of-the-art.
• On ImageNet-2012 using VGG-16, the method achieved a 1.95% accuracy gain over the full-precision model with 8-bit fixed-point quantization.
• For automatic speech recognition on the TIMIT dataset, quantization improved accuracy by 4.23% using the Deep Speech model.
• Memory usage was reduced by 3.5x to 6.4x across all three applications, demonstrating significant compression efficiency.
• Optimal performance was achieved at median bit-widths (e.g., 6–7 bits), not at the lowest or highest bit-widths, indicating a trade-off between regularization and representational capacity.
• The method reduced overfitting in large models, but degraded performance in small models, confirming its role as a regularization mechanism.