[论文解读] ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks

Ahmed T. Elthakeb, Prannoy Pilligundla+3 more|arXiv (Cornell University)|Nov 5, 2018

Advanced Neural Network Applications参考文献 48被引用 46

一句话总结

ReLeQ 使用近端策略优化（PPO）自动为深度神经网络（DNN）的层分配异质、低于8位的量化，起始于全精度模型，在保持接近原始准确度的同时实现显著的硬件加速和能耗下降。

ABSTRACT

Deep Neural Networks (DNNs) typically require massive amount of computation resource in inference tasks for computer vision applications. Quantization can significantly reduce DNN computation and storage by decreasing the bitwidth of network encodings. Recent research affirms that carefully selecting the quantization levels for each layer can preserve the accuracy while pushing the bitwidth below eight bits. However, without arduous manual effort, this deep quantization can lead to significant accuracy loss, leaving it in a position of questionable utility. As such, deep quantization opens a large hyper-parameter space (bitwidth of the layers), the exploration of which is a major challenge. We propose a systematic approach to tackle this problem, by automating the process of discovering the quantization levels through an end-to-end deep reinforcement learning framework (ReLeQ). We adapt policy optimization methods to the problem of quantization, and focus on finding the best design decisions in choosing the state and action spaces, network architecture and training framework, as well as the tuning of various hyperparamters. We show how ReLeQ can balance speed and quality, and provide an asymmetric general solution for quantization of a large variety of deep networks (AlexNet, CIFAR-10, LeNet, MobileNet-V1, ResNet-20, SVHN, and VGG-11) that virtually preserves the accuracy (=< 0.3% loss) while minimizing the computation and storage cost. With these DNNs, ReLeQ enables conventional hardware to achieve 2.2x speedup over 8-bit execution. Similarly, a custom DNN accelerator achieves 2.0x speedup and energy reduction compared to 8-bit runs. These encouraging results mark ReLeQ as the initial step towards automating the deep quantization of neural networks.

研究动机与目标

• Automate discovery of per-layer quantization levels (below 8 bits) to preserve accuracy.
• Explore how heterogenous (layer-wise) bitwidths affect overall network performance.
• Demonstrate end-to-end RL framework that balances accuracy with compute and memory cost.
• Provide a practical, hardware-agnostic method that works with conventional CPUs and custom accelerators.

提出的方法

• Formulate layer-wise quantization level selection as a multi-objective RL problem with accuracy prioritized over quantization cost.
• Use an LSTM-based policy and a value network within Proximal Policy Optimization (PPO) to learn bitwidth selection sequentially across layers.
• Define state embeddings that include static layer features and dynamic metrics such as current bitwidths and relative accuracy.
• Employ an asymmetric reward shaping that heavily penalizes accuracy loss while encouraging bitrate reduction (State Quantization).
• Train the agent starting from a full-precision model and perform short or long retraining depending on network depth to evaluate quantized performance.
• Quantize weights using WRPN-style mid-tread quantization and evaluate per-layer bitwidths in a per-layer granularity setting.

实验结果

研究问题

• RQ1 Can an RL agent autonomously discover heterogeneous per-layer bitwidths that preserve accuracy while reducing compute and storage?
• RQ2 How do layer-level bitwidth decisions interact across layers, and can an RL framework capture this interplay?
• RQ3 What reward design best guides the agent to converge to Pareto-optimal quantization patterns?
• RQ4 What hardware and software performance gains (speedup, energy reduction) are achievable with the discovered quantization patterns?

主要发现

网络	数据集	量化位宽	平均位宽	准确度损失 (%)
AlexNet	ImageNet	{8,4,4,4,4,4,4,8}	5	0.08
SimpleNet	CIFAR10	{5,5,5,5,5}	5	0.30
LeNet	MNIST	{2,2,3,2}	2.25	0.00
MobileNet	ImageNet	{8,5,6,6,4,4,7,8,4,6,8,5,5,8,6,7,7,7,6,8,6,8,8,6,7,5,5,7,8,8}	6.43	0.26
ResNet-20	CIFAR10	{8,2,2,3,2,2,2,3,2,3,3,3,2,2,2,3,2,2,2,2,2,8}	2.81	0.12
SVHN-10	SVHN	{8,4,4,4,4,4,4,4,4,8}	4.80	0.00
VGG-11	CIFAR10	{8,5,8,5,6,6,6,6,8}	6.44	0.17
VGG-16	CIFAR10	{8,8,8,6,8,6,8,6,8,6,8,6,8,6,8,8}	7.25	0.10

• ReLeQ achieves sub-8-bit per-layer quantization with accuracy loss ≤ 0.3% across a range of networks (e.g., AlexNet, LeNet, MobileNet, ResNet-20, VGG-11).
• Average bitwidths produced by ReLeQ vary by network (e.g., MobileNet 6.43 bits; ResNet-20 2.81 bits), showing heterogeneity across layers.
• On conventional hardware (TVM on CPU), ReLeQ yields ~2.2× speedup over 8-bit runs; on a custom Stripes accelerator, ~2.0× speedup and ~2.0× energy reduction over 8-bit baselines.
• Compared to ADMM-based quantization, ReLeQ provides higher performance and energy benefits across tested scenarios.
• Reward formulation strongly impacts convergence speed and final accuracy, with the proposed asymmetric reward achieving faster convergence and better relative accuracy.
• Pareto frontier analyses for several networks show ReLeQ finds solutions on the desirable region of the frontier, close to recoverable accuracy.