[Paper Review] Deep Neural Network Compression with Single and Multiple Level Quantization

Yuhui Xu, Yongzhuang Wang+3 more|arXiv (Cornell University)|Mar 6, 2018

Advanced Image and Video Retrieval Techniques43 citations

TL;DR

The paper introduces single-level quantization (SLQ) for high-bit quantization and multi-level quantization (MLQ) for extremely low-bit (ternary) quantization, leveraging width and depth partitioning to reduce model size while preserving accuracy.

ABSTRACT

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary).We are the first to consider the network quantization from both width and depth level. In the width level, parameters are divided into two parts: one for quantization and the other for re-training to eliminate the quantization loss. SLQ leverages the distribution of the parameters to improve the width level. In the depth level, we introduce incremental layer compensation to quantize layers iteratively which decreases the quantization loss in each iteration. The proposed approaches are validated with extensive experiments based on the state-of-the-art neural networks including AlexNet, VGG-16, GoogleNet and ResNet-18. Both SLQ and MLQ achieve impressive results.

Motivation & Objective

• Motivate the need for efficient DNN compression due to large parameter counts and deployment on resource-constrained devices.
• Propose two quantization strategies (SLQ and MLQ) that account for weight distribution and layer-wise importance.
• Develop a practical framework with clustering, loss-based partitioning, weight sharing, and retraining.
• Demonstrate effectiveness on standard architectures (AlexNet, VGG-16, GoogleNet, ResNet-18) and datasets (ImageNet, CIFAR-10).

Proposed method

• Use layer-wise k-means clustering to form codebooks for weights.
• Apply a loss-based partition that splits clusters into quantized and retrained groups.
• Quantize the chosen group by assigning weights to cluster centroids and retrain the remaining weights.
• Iterate clustering, partitioning, quantization, and retraining until all weights are quantized.
• Extend SLQ with ESLQ to constrain centroids toward specific value types (e.g., powers of two).
• Introduce incremental layer compensation (ILC) in MLQ to quantize layers with high loss first and then refine others, enabling ternary quantization.

Experimental results

Research questions

• RQ1 Can weight distribution-aware quantization (via clustering and loss-based partition) preserve accuracy at high bit-widths (e.g., 5 bits)?
• RQ2 Can a depth-aware strategy (ILC) enable reliable extremely low-bit (ternary) quantization without changing network architecture?
• RQ3 Does incorporating layer-wise and centerwise constraints improve quantization performance across architectures (AlexNet, VGG-16, GoogleNet, ResNet-18)?
• RQ4 How do SLQ and MLQ compare to existing quantization methods (e.g., INQ, TWN, TTQ) on benchmarks like ImageNet and CIFAR-10?

Key findings

• 5-bit SLQ on ImageNet-derived networks (e.g., VGG-16) achieves higher Top-1/Top-5 accuracy than the full-precision reference in some cases.
• On ImageNet, 5-bit SLQ shows improvements over INQ for several networks (e.g., VGG-16: Top-1 72.23% vs 68.54% reference; Top-5 91.0% vs 88.65%).
• 4-bit SLQ and 3-bit SLQ results indicate maintained or slightly degraded accuracy depending on centroid count (e.g., VGG-16 4-bit: Top-1 71.18%, Top-5 90.25%).
• MLQ enables ternary quantization with minimal accuracy loss on CIFAR-10 (Light CNN: 78.46% vs 78.66% in Top-1; ResNet20: 90.02% vs 91.70%); AlexNet on ImageNet achieves comparable results without adding BN layers.
• MLQ on ImageNet without BN achieves 54.24% Top-1 and 77.78% Top-5 (no BN) illustrating competitive performance without architecture changes.
• 5-bit AlexNet with SLQ achieves ~6x compression; combining with pruning can reach ~53x without accuracy loss.