[Paper Review] HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision

Zhen Dong, Zhewei Yao+3 more|arXiv (Cornell University)|Apr 29, 2019

Advanced Neural Network Applications41 references36 citations

TL;DR

HAWQ uses Hessian-based second-order information to automatically assign layer-wise mixed-precision and a deterministic fine-tuning order, enabling higher compression with comparable or better accuracy across CIFAR-10 and ImageNet models.

ABSTRACT

Model size and inference speed/power have become a major challenge in the deployment of Neural Networks for many applications. A promising approach to address these problems is quantization. However, uniformly quantizing a model to ultra low precision leads to significant accuracy degradation. A novel solution for this is to use mixed-precision quantization, as some parts of the network may allow lower precision as compared to other layers. However, there is no systematic way to determine the precision of different layers. A brute force approach is not feasible for deep networks, as the search space for mixed-precision is exponential in the number of layers. Another challenge is a similar factorial complexity for determining block-wise fine-tuning order when quantizing the model to a target precision. Here, we introduce Hessian AWare Quantization (HAWQ), a novel second-order quantization method to address these problems. HAWQ allows for the automatic selection of the relative quantization precision of each layer, based on the layer's Hessian spectrum. Moreover, HAWQ provides a deterministic fine-tuning order for quantizing layers, based on second-order information. We show the results of our method on Cifar-10 using ResNet20, and on ImageNet using Inception-V3, ResNet50 and SqueezeNext models. Comparing HAWQ with state-of-the-art shows that we can achieve similar/better accuracy with $8 imes$ activation compression ratio on ResNet20, as compared to DNAS~\cite{wu2018mixed}, and up to $1\%$ higher accuracy with up to $14\%$ smaller models on ResNet50 and Inception-V3, compared to recently proposed methods of RVQuant~\cite{park2018value} and HAQ~\cite{wang2018haq}. Furthermore, we show that we can quantize SqueezeNext to just 1MB model size while achieving above $68\%$ top1 accuracy on ImageNet.

Motivation & Objective

• Motivate reducing model size and inference energy via quantization without uniform precision across layers.
• Propose a Hessian-based method to automatically determine per-layer quantization precision.
• Provide a deterministic, second-order informed fine-tuning order for quantized models.
• Demonstrate state-of-the-art or competitive accuracy at high compression on CIFAR-10 and ImageNet.

Proposed method

• Partition network into blocks and compute Hessian eigenvalues for each block via a matrix-free power iteration (Hessian matvec).
• Define a per-block sensitivity metric S_i = lambda_i / n_i to rank precision choices (lambda_i top Hessian eigenvalue, n_i block size).
• Assign relative bit-precision order for blocks based on descending S_i to guide mixed-precision quantization.
• Define a fine-tuning order using Omega_i = lambda_i * ||Q(W_i) - W_i||^2 to prioritize blocks with high curvature and large perturbations.
• Approximate the Hessian as a block-diagonal matrix scaled by top eigenvalues to reduce complexity.
• Perform block-wise, Hessian-informed quantization and subsequent block-wise fine-tuning in the derived order.
• Provide an ablation study showing the impact of ordering and Hessian guidance on performance.

Experimental results

Research questions

• RQ1 Can Hessian-spectrum information guide the selection of relative per-layer/block quantization precision in mixed-precision quantization?
• RQ2 Can Hessian-guided ordering of block-wise quantization and subsequent fine-tuning achieve higher compression with comparable accuracy to state-of-the-art methods?
• RQ3 What is the impact of Hessian-based ordering on convergence speed and final accuracy for large vision models?
• RQ4 How does HAWQ perform across CIFAR-10 with ResNet20 and ImageNet with Inception-V3, ResNet-50, and SqueezeNext compared to existing quantization methods?

Key findings

• HAWQ achieves similar or better accuracy with substantially higher compression, particularly in activations, on CIFAR-10 ResNet20 (up to 8x activation compression).
• On ImageNet, HAWQ attains up to 12.04x weight+activation compression for Inception-V3 with Top-1 around 75.52% (vs Direct) and outperforms several alternatives in both accuracy and model size.
• For ResNet-50 on ImageNet, HAWQ reaches 75.48% Top-1 with 7.96 MB model size, outperforming HAQ in both accuracy and size.
• For SqueezeNext on ImageNet, HAWQ reduces to 1 MB with 68.02% Top-1 accuracy, greatly surpassing direct quantization and enabling ultra-compact deployment.
• Ablation shows Hessian-guided ordering (both for quantization and fine-tuning) substantially improves convergence speed and final accuracy compared to reversed-order baselines.
• The approach demonstrates that second-order information provides a principled, scalable way to perform mixed-precision quantization across diverse architectures.