[Paper Review] Quantizing deep convolutional networks for efficient inference: A whitepaper

Raghuraman Krishnamoorthi|arXiv (Cornell University)|Jun 21, 2018

Advanced Neural Network ApplicationsComputer Science756 citations

TL;DR

This whitepaper surveys post-training and quantization-aware methods to quantize CNN weights and activations to 8, 4, or 16 bits, analyzes accuracy impact across architectures, and provides TensorFlow tooling and training best practices for efficient inference on edge devices.

ABSTRACT

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

Motivation & Objective

• Motivate quantization to enable edge inference with reduced model size, memory, and power.
• Characterize accuracy and latency trade-offs for 8-, 4-, and 16-bit quantization across multiple CNN architectures.
• Develop and assess both post-training quantization and quantization-aware training approaches.
• Provide practical guidelines and tooling for implementing quantized models in TensorFlow/TensorFlow Lite.

Proposed method

• Present uniform affine, symmetric, and stochastic quantizers for weights and activations.
• Derive forward and backward pass formulations under simulated quantization and straight-through estimators.
• Evaluate post-training quantization (weight-only and weight-activation) and quantization-aware training across networks like MobileNet, Inception, ResNet, NasNet on ImageNet.
• Analyze the impact of quantization granularity (per-layer vs per-channel) on accuracy.
• Discuss batch normalization handling and compute-friendly folding strategies for quantized inference.
• Offer TensorFlow tooling and a practical workflow for quantization and deployment.

Experimental results

Research questions

• RQ1 What quantization schemes (per-channel weights, per-layer activations) preserve accuracy close to floating-point across common CNN architectures?
• RQ2 How do post-training quantization and quantization-aware training compare in terms of accuracy at 8-bit and lower bitwidths?
• RQ3 What are the effects of batch normalization handling on quantized CNN accuracy and stability?
• RQ4 What tooling and training workflows enable practical deployment of quantized models in TensorFlow/TensorFlow Lite?
• RQ5 What are the latency and model-size benefits for CPUs, DSPs, and specialized accelerators when using quantized networks?

Key findings

• Per-channel weight quantization with per-layer activations at 8-bit accuracy yields results within about 2% of FP32 for many architectures.
• Weight quantization to 8 bits reduces model size by ~4x even when 8-bit arithmetic is unavailable, via post-training quantization.
• Quantized networks run faster on CPUs (2x–3x) and substantially faster on fixed-point SIMD hardware (up to ~10x).
• Quantization-aware training narrows the FP32 gap to about 1% at 8-bit precision and enables 4-bit weight quantization with 2%–10% accuracy loss depending on architecture.
• Per-channel weight quantization generally outperforms per-layer when quantizing weights, especially at 4-bit precision with fine-tuning.
• Activation quantization to 8-bit incurs little accuracy loss due to normalization strategies like BatchNorm simplifications or ReLU6.