[Paper Review] AutoQ: Automated Kernel-Wise Neural Network Quantization

Qian Lou, Feng Guo+3 more|arXiv (Cornell University)|Feb 15, 2019

Advanced Neural Network Applications34 references22 citations

TL;DR

AutoQ proposes a hierarchical deep reinforcement learning (DRL) framework that automatically searches optimal quantization bitwidths (QBN) for individual weight kernels and activation layers in convolutional neural networks, significantly reducing inference latency and energy consumption while maintaining accuracy. By using a high-level controller to set target QBNs and a low-level controller to optimize per-kernel quantization, AutoQ achieves 54.06% lower latency and 50.69% lower energy consumption compared to state-of-the-art DRL-based methods.

ABSTRACT

Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06\%, and decrease the inference energy consumption by 50.69\%, while achieving the same inference accuracy.

Motivation & Objective

• To address the challenge of inefficient kernel-wise neural network quantization due to the prohibitively large search space for per-kernel bitwidth selection.
• To overcome the limitations of existing DRL-based methods that struggle with convergence and suboptimal configurations in kernel-wise quantization.
• To enable accurate, low-latency, and energy-efficient deployment of CNNs on low-power mobile devices by automatically determining optimal QBNs for each weight kernel and activation layer.
• To balance the trade-off between inference accuracy, latency, energy consumption, and FPGA area through a novel extrinsic reward function.

Proposed method

• AutoQ employs a hierarchical DRL architecture with a high-level controller (HLC) that sets target average QBNs for weight kernels and activation layers.
• The low-level controller (LLC) generates per-kernel QBNs based on the HLC’s goals, enabling fine-grained, kernel-wise quantization.
• A shaped intrinsic reward is introduced to accelerate learning by combining HLC goal completion and extrinsic environment feedback.
• The extrinsic reward function balances inference accuracy, latency, energy consumption, and FPGA area using weighted coefficients (ψ_acc=2, ψ_l=0.5, ψ_e=0.5, ψ_a=0.5).
• The method is evaluated on both temporal and spatial CNN accelerators, including BitFusion and FPGA-based systems, to validate hardware efficiency.

Experimental results

Research questions

• RQ1 Can a hierarchical DRL framework effectively explore the large search space of per-kernel quantization bitwidths while maintaining high inference accuracy?
• RQ2 How does the shaped intrinsic reward improve learning efficiency compared to standard DRL agents like HIRO or DDPG?
• RQ3 To what extent can kernel-wise quantization reduce inference latency and energy consumption on both temporal and spatial CNN accelerators?
• RQ4 Can the extrinsic reward function effectively balance multiple hardware and accuracy metrics during the search process?
• RQ5 Does sub-kernel-wise quantization provide further benefits on systolic array-based spatial accelerators?

Key findings

• AutoQ reduces inference latency by 54.06% and energy consumption by 50.69% on average compared to state-of-the-art DRL-based quantization methods, while maintaining the same accuracy.
• The hierarchical DRL approach reaches ~70% inference accuracy in only 200 episodes, significantly outperforming DDPG-based methods that remain stuck at ~20% accuracy until 250 episodes.
• On a spatial CNN accelerator (BitFusion), kernel-wise quantization reduces latency by 39.04% and energy by 33.34% compared to layer-wise quantization.
• Sub-kernel-wise quantization does not improve performance on systolic array accelerators, as the largest QBN in a kernel still dictates the hardware resource usage.
• The extrinsic reward enables AutoQ to identify a QBN configuration that uses 85% FPGA area over a competing one using 94%, demonstrating effective hardware resource optimization.