[Paper Review] EasyQuant: Post-training Quantization via Scale Optimization

Di Wu, Qi Tang+4 more|arXiv (Cornell University)|Jun 30, 2020

Advanced Neural Network Applications31 references39 citations

TL;DR

EasyQuant (EQ) performs post-training quantization by alternately optimizing per-layer scales for weights and activations, achieving near-INT8 accuracy with 7-bit quantization and offering INT7 inference on ARM with improved latency over TRT.

ABSTRACT

The 8 bits quantization has been widely applied to accelerate network inference in various deep learning applications. There are two kinds of quantization methods, training-based quantization and post-training quantization. Training-based approach suffers from a cumbersome training process, while post-training quantization may lead to unacceptable accuracy drop. In this paper, we present an efficient and simple post-training method via scale optimization, named EasyQuant (EQ),that could obtain comparable accuracy with the training-based method.Specifically, we first alternately optimize scales of weights and activations for all layers target at convolutional outputs to further obtain the high quantization precision. Then, we lower down bit width to INT7 both for weights and activations, and adopt INT16 intermediate storage and integer Winograd convolution implementation to accelerate inference.Experimental results on various computer vision tasks show that EQ outperforms the TensorRT method and can achieve near INT8 accuracy in 7 bits width post-training.

Motivation & Objective

• Motivate why post-training quantization is preferable to retraining in constrained environments.
• Propose a simple, efficient scale-optimization method to maximize similarity between FP32 and quantized convolution outputs.
• Show that alternating optimization of weight and activation scales yields competitive accuracy with training-based quantization.
• Demonstrate INT7 post-training inference with INT16 storage and integer Winograd to improve hardware latency on ARM platforms.
• Validate the method across image classification, object detection, and face recognition tasks.

Proposed method

• Model quantization is formulated as Q(X,S)=Clip(Round(X·S)) with layer-wise scales for activations (S^a) and weights (S^w).
• For each layer, optimize S^w (with S^a fixed) and then S^a (with S^w fixed) to maximize the cosine similarity between FP32 outputs O_l and quantized outputs Ō_l over a calibration set.
• Sequential, greedy layer-wise optimization is applied across the network, iterating until convergence or time limit.
• Initialization uses the maximum values of weights and activations; a simple linear search over candidate scales within [0.5·S_l, 2·S_l] with 100 candidates is used per layer.
• For per-channel quantization, individual scales can be tuned per filter in parallel.
• INT7 post-training inference is implemented on ARM using INT16 intermediates and Winograd-based convolution to improve speed and reduce storage compared to INT8 workflows.

Experimental results

Research questions

• RQ1 Can post-training quantization reach accuracy close to training-based quantization by optimizing per-layer scales for weights and activations?
• RQ2 Does alternating optimization of activation and weight scales per layer improve the cosine similarity between FP32 and quantized convolution outputs enough to sustain accuracy across tasks?
• RQ3 Is INT7 post-training inference feasible with practical latency improvements on ARM devices using INT16 intermediates and integer Winograd?
• RQ4 How does EasyQuant perform across image classification, object detection, and face recognition benchmarks compared to TensorRT and QAT baselines?
• RQ5 What are the limitations of 7-bit quantization in vision tasks, and how does EQ mitigate them?

Key findings

• EQ achieves competitive or better accuracy than TensorRT (TRT) in INT8 per-channel quantization across multiple models and tasks.
• EQ attains near-INT8 accuracy in 7-bit quantization (INT7) on several architectures and tasks, with better robustness than TRT in INT7 scenarios.
• INT7 inference with INT16 intermediates and integer Winograd yields lower latency on ARM platforms compared to INT8 baselines.
• Across ImageNet classification, VOC2007 object detection, and standard face recognition benchmarks, EQ consistently outperforms TRT in INT8/INT7 settings.
• CASES show EQ can be competitive with or exceed results from QAT on MobileNetV1 and ResNet50 in INT8 quantization.
• INT7 deployment benefits from faster SMLAL/SADALP-based computations and reduced memory traffic.