[Paper Review] A Survey on Methods and Theories of Quantized Neural Networks

Yunhui Guo|arXiv (Cornell University)|Aug 13, 2018

Advanced Neural Network ApplicationsComputer Science77 references204 citations

TL;DR

This survey comprehensively reviews quantized neural networks, covering quantization techniques (deterministic, stochastic, probabilistic), methods for quantizing weights, activations, and gradients, optimization frameworks, and practical challenges with training and deployment.

ABSTRACT

Deep neural networks are the state-of-the-art methods for many real-world tasks, such as computer vision, natural language processing and speech recognition. For all its popularity, deep neural networks are also criticized for consuming a lot of memory and draining battery life of devices during training and inference. This makes it hard to deploy these models on mobile or embedded devices which have tight resource constraints. Quantization is recognized as one of the most effective approaches to satisfy the extreme memory requirements that deep neural network models demand. Instead of adopting 32-bit floating point format to represent weights, quantized representations store weights using more compact formats such as integers or even binary numbers. Despite a possible degradation in predictive performance, quantization provides a potential solution to greatly reduce the model size and the energy consumption. In this survey, we give a thorough review of different aspects of quantized neural networks. Current challenges and trends of quantized neural networks are also discussed.

Motivation & Objective

• Assess the motivation and impact of quantizing neural networks to reduce memory and energy usage.
• Categorize and compare quantization techniques (deterministic, stochastic, probabilistic) and their applications to weights, activations, and gradients.
• Discuss optimization formulations and training strategies that enable effective quantization.
• Summarize practical methods for quantizing different network components and hardware considerations.

Proposed method

• Classify quantization techniques into deterministic, stochastic, and probabilistic approaches.
• Explain weight, activation, and gradient quantization and their respective challenges.
• Present optimization formulations (e.g., J(B,α)=||W−αB||^2, ADMM-based approaches, loss-aware binarization).
• Describe codebooks (fixed vs adaptive) and common quantization schemes (binary, ternary, power-of-two).
• Discuss training strategies (STE, proximal Newton, ADMM, variational inference) and their convergence considerations.
• Provide a hardware-oriented comparison of methods with respect to precision, efficiency, and deployment.

Experimental results

Research questions

• RQ1 What are the main quantization paradigms used for neural networks and how do they differ in practice?
• RQ2 How can weights, activations, and gradients be quantized while preserving performance?
• RQ3 What optimization frameworks effectively train quantized networks from scratch or by post-training quantization?
• RQ4 What are the practical challenges in training and deploying quantized models on hardware?

Key findings

• Binary networks can achieve high accuracy on MNIST (example: 98.8% on MNIST with binary weights/activations).
• Ternary networks can attain comparable performance to full-precision networks on large datasets like ImageNet in some cases.
• Several optimization formulations (e.g., XNOR-net, loss-aware binarization, ADMM-based methods) enable effective quantization with maintained accuracy.
• Vector quantization and product quantization offer aggressive compression but are typically applied to pre-trained models rather than training-from-scratch scenarios.
• Probabilistic quantization and Bayesian perspectives yield sparse models and interpretable weight distributions, albeit with tractable modeling challenges and potential limitations for some architectures.
• Deterministic quantization is favored for hardware accelerations and fixed codebooks, while stochastic/probabilistic methods offer regularization and interpretability.