[Paper Review] Towards the Limit of Network Quantization

Yoojin Choi, Mostafa El‐Khamy+1 more|arXiv (Cornell University)|Dec 5, 2016

Advanced Image and Video Retrieval Techniques25 references62 citations

TL;DR

This paper proposes Hessian-weighted k-means clustering and entropy-constrained scalar quantization (ECSQ) for deep neural network quantization to minimize performance loss under compression ratio constraints. By leveraging the Hessian matrix to weight quantization errors and using Huffman coding, the method achieves compression ratios of 51.25× (LeNet), 22.17× (ResNet), and 40.65× (AlexNet) with negligible accuracy drop.

ABSTRACT

Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.

Motivation & Objective

• Address the suboptimality of conventional k-means clustering in network quantization, which ignores the impact of quantization errors on network loss.
• Minimize performance loss due to quantization under a fixed compression ratio constraint by modeling the error impact via the Hessian matrix.
• Establish a theoretical link between network quantization and entropy-constrained scalar quantization (ECSQ) in information theory for optimal variable-length coding.
• Enable joint quantization of all network layers simultaneously, avoiding layer-by-layer optimization and improving compression efficiency.
• Provide a practical, low-cost alternative to Hessian computation using gradient second-moment estimates from Adam optimizer for real-world deployment.

Proposed method

• Formulate network quantization as minimizing Hessian-weighted distortion, where the Hessian matrix measures parameter sensitivity to loss changes.
• Propose Hessian-weighted k-means clustering to group network parameters such that high-impact parameters are quantized with lower error.
• Reframe the quantization problem under compression ratio constraints as an entropy-constrained scalar quantization (ECSQ) problem when using optimal variable-length codes like Huffman coding.
• Introduce two heuristic solutions for ECSQ: uniform quantization and an iterative algorithm inspired by Lloyd’s algorithm for scalar quantization.
• Use the square root of second-moment estimates of gradients as a low-cost Hessian approximation during training, available at no extra cost with Adam optimizer.
• Apply joint quantization across all network layers, enabling global optimization without per-layer compression rate tuning.

Experimental results

Research questions

• RQ1 How can quantization error be optimally minimized in deep neural networks under a fixed compression ratio constraint?
• RQ2 What is the appropriate weighting metric for quantization error that reflects its impact on network loss?
• RQ3 Can the network quantization problem be formally connected to information-theoretic data compression problems like ECSQ?
• RQ4 How does joint quantization of all layers compare to layer-by-layer quantization in terms of performance and compression efficiency?
• RQ5 Can a low-cost alternative to the Hessian matrix be used effectively in quantization without sacrificing accuracy?

Key findings

• Hessian-weighted k-means clustering reduces performance loss more effectively than standard k-means, especially when using fixed-length coding.
• When combined with Huffman coding, uniform quantization and the iterative ECSQ algorithm outperform Hessian-weighted k-means, as they are optimized for variable-length codes.
• Using only 1,000 samples to estimate the Hessian yields performance nearly identical to full-batch estimation, enabling efficient computation.
• The square root of second-moment gradient estimates serves as a highly effective, low-cost alternative to the Hessian, matching its performance with no additional training cost.
• Joint quantization of all layers together achieves superior compression ratios—51.25× for LeNet, 22.17× for 32-layer ResNet, and 40.65× for AlexNet—while maintaining near-original accuracy.
• The proposed method achieves higher compression ratios than prior work (e.g., Han et al., 2015a), with 51.25× vs. 39.00× for LeNet and 22.17× vs. N/A for ResNet under comparable settings.