[Paper Review] HAQ: Hardware-Aware Automated Quantization

Kuan Wang, Zhijian Liu+3 more|arXiv (Cornell University)|Nov 21, 2018

Advanced Neural Network Applications22 references58 citations

TL;DR

HAQ proposes a reinforcement learning-based framework that automatically determines optimal mixed-precision quantization policies for DNNs by incorporating real-time hardware feedback (latency and energy) via a hardware simulator. It achieves 1.4–1.95× latency reduction and 1.9× energy savings over 8-bit quantization with minimal accuracy loss, while revealing distinct optimal policies across edge and cloud hardware architectures.

ABSTRACT

Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.

Motivation & Objective

• Address the challenge of manually exploring the vast design space of mixed-precision quantization across diverse hardware accelerators.
• Overcome the limitations of conventional quantization methods that apply uniform bitwidths and ignore hardware-specific performance characteristics.
• Enable automated, hardware-specialized quantization policies that optimize for latency, energy, model size, and accuracy simultaneously.
• Provide insights into how different hardware architectures (e.g., edge vs. cloud) require fundamentally different quantization strategies under varying resource constraints.

Proposed method

• Employ a reinforcement learning agent to search for optimal per-layer bitwidths (1–8 bits) across DNN layers.
• Integrate a hardware simulator into the RL loop to provide direct feedback on latency and energy consumption, replacing proxy metrics like FLOPs.
• Train the RL agent using a reward function that balances model accuracy, latency, energy efficiency, and model size.
• Use a policy gradient method (e.g., REINFORCE or PPO) to optimize the quantization policy based on hardware feedback.
• Support end-to-end automation by co-designing the quantization policy with the target hardware accelerator.
• Enable transferability of learned policies across different neural network architectures by training on a diverse set of models.

Experimental results

Research questions

• RQ1 How does hardware-aware quantization via RL improve inference efficiency compared to fixed-precision quantization?
• RQ2 What are the differences in optimal quantization policies between edge and cloud hardware architectures under varying resource constraints?
• RQ3 To what extent can hardware feedback from a simulator replace proxy metrics like FLOPs in guiding quantization decisions?
• RQ4 How does the proposed framework maintain high model accuracy while reducing latency and energy consumption?

Key findings

• HAQ reduced inference latency by 1.4× to 1.95× compared to 8-bit fixed-precision quantization across multiple models and hardware platforms.
• Energy consumption was reduced by 1.9× on average, demonstrating significant efficiency gains without substantial accuracy degradation.
• The optimal quantization policy varied drastically between edge and cloud hardware, indicating that one-size-fits-all quantization is suboptimal.
• Hardware-specific feedback from a simulator enabled the RL agent to discover more efficient bitwidth allocations than those based on proxy metrics.
• The framework revealed that mixed-precision policies are highly sensitive to target hardware characteristics, offering actionable insights for both hardware and neural architecture co-design.
• The method achieved near-identical accuracy to full-precision models while significantly reducing model size and computational cost.