[Paper Review] Ternary Neural Networks with Fine-Grained Quantization

Naveen Mellempudi, Abhisek Kundu+4 more|arXiv (Cornell University)|May 2, 2017

Advanced Neural Network Applications11 references61 citations

TL;DR

FGQ converts pre-trained full-precision models to ternary weights with 8/4-bit activations without retraining, using weight groups to balance accuracy and compute reductions; achieves near FP32 accuracy on ImageNet with significant speedups.

ABSTRACT

We propose a novel fine-grained quantization (FGQ) method to ternarize pre-trained full precision models, while also constraining activations to 8 and 4-bits. Using this method, we demonstrate a minimal loss in classification accuracy on state-of-the-art topologies without additional training. We provide an improved theoretical formulation that forms the basis for a higher quality solution using FGQ. Our method involves ternarizing the original weight tensor in groups of $N$ weights. Using $N=4$, we achieve Top-1 accuracy within $3.7\%$ and $4.2\%$ of the baseline full precision result for Resnet-101 and Resnet-50 respectively, while eliminating $75\%$ of all multiplications. These results enable a full 8/4-bit inference pipeline, with best-reported accuracy using ternary weights on ImageNet dataset, with a potential of $9 imes$ improvement in performance. Also, for smaller networks like AlexNet, FGQ achieves state-of-the-art results. We further study the impact of group size on both performance and accuracy. With a group size of $N=64$, we eliminate $\approx99\%$ of the multiplications; however, this introduces a noticeable drop in accuracy, which necessitates fine tuning the parameters at lower precision. We address this by fine-tuning Resnet-50 with 8-bit activations and ternary weights at $N=64$, improving the Top-1 accuracy to within $4\%$ of the full precision result with $<30\%$ additional training overhead. Our final quantized model can run on a full 8-bit compute pipeline using 2-bit weights and has the potential of up to $15 imes$ improvement in performance compared to baseline full-precision models.

Motivation & Objective

• Motivate and enable near state-of-the-art inference with extremely low-precision weights and activations without (or with minimal) retraining.
• Introduce a fine-grained quantization (FGQ) method that ternarizes pre-trained weights in groups to preserve information.
• Show that FGQ achieves high Top-1 accuracy on ImageNet with ResNet-101/ResNet-50 and AlexNet using 2w-8a and 2w-4a.
• Analyze the impact of group size on accuracy and compute savings, and discuss hardware implications for 8-bit compute pipelines.

Proposed method

• Ternarize the full-precision weight tensor in disjoint groups of size N, producing independent sub-problems for each group.
• For each group, solve for alpha and a ternarized weight vector to minimize ||W^(i) - alpha * W^^(i)||_F^2 (Eq. 2).
• Use separate thresholds Delta_p and Delta_n for positive/negative weights with a single alpha, solving for alpha*, Delta_p*, Delta_n* via closed-form or brute-force (Eqs. 3–5).
• Adopt a static grouping strategy along the input channel dimension to minimize within-group dynamic range and enable efficient memory layout and vectorization (Fig. 2).
• Quantize activations to 8/4 bits and apply 32-bit accumulators during computation to prevent overflow; recompute batch-norm statistics during inference to compensate for variance shifts.
• Experiment with group sizes (N) to trade accuracy versus percentage of ternary-accumulations (e.g., N=4 yields 75% ternary FPAs, N=64 yields ~99%).

Experimental results

Research questions

• RQ1 Can pre-trained full-precision networks be converted to ternary weights with minimal loss in accuracy without retraining?
• RQ2 How does fine-grained grouping (N) affect accuracy versus compute savings in a 2w-8a/2w-4a inference pipeline?
• RQ3 What is the best grouping strategy for preserving distributions of weights across layers to maximize accuracy?
• RQ4 Can FGQ achieve state-of-the-art or near state-of-the-art accuracy on ImageNet with ResNet-101, ResNet-50, and AlexNet without retraining?
• RQ5 What are the practical hardware implications and performance gains of FGQ for full 8-bit compute pipelines?

Key findings

• FGQ with N=4 (FGQ-N4) achieves Top-1 accuracy of 73.85% on ResNet-101 with 2w-8a and 70.69% with 2w-4a on ImageNet, without re-training.
• FGQ-N4 applied to ResNet-50 yields 70.76% Top-1 with 2w-8a and 68.38% with 2w-4a, near full-precision results.
• FGQ-N4 applied to AlexNet yields 49.04% Top-1 with 2w-8a (no re-training), about 8% away from the baseline 56.83%.
• Larger group sizes (e.g., N=64) can eliminate ~99% of multiplications but cause noticeable accuracy loss, which can be mitigated by limited low-precision retraining.
• The approach enables a full 8-bit compute pipeline with 2-bit weights and up to 15x theoretical performance improvement over full-precision baselines.
• Compared to closely related works, FGQ achieves competitive or superior accuracy without low-precision training for many configurations.