[Paper Review] Trained Ternary Quantization

Chenzhuo Zhu, Song Han+2 more|arXiv (Cornell University)|Dec 4, 2016

Advanced Neural Network Applications15 references746 citations

TL;DR

Trained Ternary Quantization (TTQ) trains per-layer ternary weights with learnable positive and negative scaling factors, achieving about 16x model size reduction with minimal accuracy loss and even slight improvements on CIFAR-10 and ImageNet.

ABSTRACT

Deep neural networks are widely used in machine learning applications. However, the deployment of large neural networks models can be difficult to deploy on mobile devices with limited power budgets. To solve this problem, we propose Trained Ternary Quantization (TTQ), a method that can reduce the precision of weights in neural networks to ternary values. This method has very little accuracy degradation and can even improve the accuracy of some models (32, 44, 56-layer ResNet) on CIFAR-10 and AlexNet on ImageNet. And our AlexNet model is trained from scratch, which means it's as easy as to train normal full precision model. We highlight our trained quantization method that can learn both ternary values and ternary assignment. During inference, only ternary values (2-bit weights) and scaling factors are needed, therefore our models are nearly 16x smaller than full-precision models. Our ternary models can also be viewed as sparse binary weight networks, which can potentially be accelerated with custom circuit. Experiments on CIFAR-10 show that the ternary models obtained by trained quantization method outperform full-precision models of ResNet-32,44,56 by 0.04%, 0.16%, 0.36%, respectively. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.

Motivation & Objective

• Motivate deployment of deep neural networks on mobile devices by reducing model size and energy consumption.
• Introduce a quantization method that learns both ternary values and ternary assignments for weights.
• Maintain a latent full-precision weight representation during training to guide quantization.
• Enable efficient inference using only ternary weights and per-layer scaling factors.
• Demonstrate improved or comparable accuracy to full-precision models on CIFAR-10 and ImageNet.

Proposed method

• Use per-layer scaling coefficients Wp_l and Wn_l to quantize weights to {+Wp_l, 0, -Wn_l}.
• Back-propagate gradients to both the latent full-precision weights and the scaling coefficients to learn ternary values and assignments.
• Quantize by thresholds Delta_l proportional to the layer’s maximum absolute weight, with a fixed factor t across layers.
• During training, back-propagate gradient to Wp_l, Wn_l using equations for index sets Ip_l and In_l to update the codebook and the latent weights.
• At inference, discard full-precision weights and use only ternary weights and scaling factors for computation.
• Explore sparsity via a fixed threshold heuristic and show layer-wise behavior of Wp_l and Wn_l over training.

Experimental results

Research questions

• RQ1 Can TTQ maintain or improve accuracy compared to full-precision networks on CIFAR-10 and ImageNet despite aggressive quantization?
• RQ2 How do trainable ternary values and asymmetric scaling factors affect model capacity and learning dynamics?
• RQ3 What are the compression and energy efficiency benefits of TTQ in practice across common architectures?
• RQ4 How does TTQ compare to prior ternary/binary quantization methods (e.g., TWN, DoReFa-Net) on standard benchmarks?

Key findings

• TTQ achieves 16x reduction in parameter size by using 2-bit ternary weights with layer-wise scaling factors.
• On CIFAR-10, TTQ improves ResNet-32/44/56 accuracy by 0.04%, 0.16%, and 0.36%, respectively, over full-precision baselines.
• On ImageNet, TTQ with AlexNet from scratch reaches 42.5% Top-1 accuracy, outperforming full-precision AlexNet by 1.6% and surpassing prior ternary models by about 0.3% in Top-1 across reported results.
• TTQ outperforms prior ternary networks (TWN) by approximately 3% Top-1 on ImageNet.
• The asymmetry in Wp_l ≠ Wn_l increases model capacity and the ternary weights act as learning-rate multipliers during backpropagation.
• Visualization shows learned ternary kernels preserve key edge/corner detectors despite low precision, indicating effective feature extraction.