[Paper Review] Training Quantized Nets: A Deeper Understanding

Hao Li, Soham De+4 more|arXiv (Cornell University)|Jun 7, 2017

Adversarial Robustness in Machine Learning10 references94 citations

TL;DR

The paper analyzes training quantized neural networks from a theoretical perspective, comparing stochastic rounding and BinaryConnect, proving convergence guarantees and explaining why fully-quantized methods struggle to greedily optimize, with experiments on CIFAR-10/100 and ImageNet.

ABSTRACT

Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.

Motivation & Objective

• Motivate and analyze training quantized neural networks directly from scratch for embedded, low-precision hardware.
• Develop theoretical convergence results for stochastic rounding (SR) and BinaryConnect (BC) under convex and non-convex settings.
• Explain why floating-point retention (BC) facilitates optimization while fully quantized methods (SR) stall.
• Compare SR and BC behavior in non-convex problems to understand exploration-exploitation dynamics.
• Provide empirical validation on standard architectures and datasets to illustrate theory.

Proposed method

• Formulate quantized training with a quantization operator Q (deterministic or stochastic) applied during updates.
• Prove convergence results for SR showing an accuracy floor proportional to the quantization level Δ in convex settings.
• Prove convergence results for BC showing improved behavior when the objective is strongly convex or quadratic, and quantify error floors.
• Analyze non-convex behavior to contrast SR’s lack of greedy exploitation with BC’s annealing-like improvement.
• Conduct experiments training VGG/ResNet variants with binarized weights using SR-ADAM, BC-ADAM, R-ADAM, and Big SR-ADAM on CIFAR-10, CIFAR-100, and ImageNet.

Experimental results

Research questions

• RQ1 Can stochastic rounding (SR) and BinaryConnect (BC) converge when training quantized networks from scratch?
• RQ2 What are the accuracy floors and convergence rates for SR and BC under convex and non-convex objectives?
• RQ3 Why does BC often outperform fully quantized methods like SR in neural network training?
• RQ4 How do SR and BC differ in exploration-exploitation dynamics during non-convex optimization?
• RQ5 Do empirical results on standard architectures align with the theoretical predictions for quantized training methods?

Key findings

• SR and BC converge to within O(Δ) accuracy of the minimizer in convex settings, with the accuracy floor depending on quantization level.
• BC can converge to true minimizers for quadratic (or near-quadratic) problems, while SR stalls at an accuracy floor independent of refined stepsizes.
• In non-convex problems, SR lacks the greedy exploitation phase that helps BC—BC’s iterates can concentrate near minimizers as learning rates shrink, whereas SR stagnates.
• Experiments show BC-ADAM matches full-precision ADAM on CIFAR-10/100 and ImageNet in some cases, while SR-ADAM and R-ADAM underperform, consistent with theory.
• Big-batch SR improves performance by increasing exploration efficiency, addressing SR’s inability to exploit local minima with small learning rates.
• SR tends to explore more weight changes than BC across networks and datasets, aligning with theoretical predictions of exploration-exploitation dynamics.