[Paper Review] Ternary Weight Networks

Fengfu Li, Bin Liu+3 more|arXiv (Cornell University)|May 16, 2016

Advanced Neural Network Applications21 references758 citations

TL;DR

This paper introduces ternary weight networks (weights in +1, 0, -1) with a scaling factor to approximate full-precision weights, achieving significant model compression and competitive accuracy on vision tasks, plus improved detection performance over binary nets.

ABSTRACT

We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1. The Euclidian distance between full (float or double) precision weights and the ternary weights along with a scaling factor is minimized in training stage. Besides, a threshold-based ternary function is optimized to get an approximated solution which can be fast and easily computed. TWNs have shown better expressive abilities than binary precision counterparts. Meanwhile, TWNs achieve up to 16$ imes$ model compression rate and need fewer multiplications compared with the float32 precision counterparts. Extensive experiments on MNIST, CIFAR-10, and ImageNet datasets show that the TWNs achieve much better result than the Binary-Weight-Networks (BWNs) and the classification performance on MNIST and CIFAR-10 is very close to the full precision networks. We also verify our method on object detection task and show that TWNs significantly outperforms BWN by more than 10\% mAP on PASCAL VOC dataset. The pytorch version of source code is available at: https://github.com/Thinklab-SJTU/twns.

Motivation & Objective

• Motivate reducing storage and computation in deep neural networks for edge devices.
• Propose a ternary weight quantization with a scaling factor to approximate full-precision weights.
• Develop a threshold-based ternary function to efficiently obtain ternary weights.
• Provide training and inference procedures that leverage ternary weights with minimal multiplications.
• Show empirical results comparing TWNs to full-precision and binary-weight networks across classification and detection tasks.

Proposed method

• Formulate objective as minimizing the Euclidean distance between full-precision weights and scaled ternary weights, minimizing ||W - αW~||2 with W~ ∈ {-1,0,1}.
• Introduce a threshold-based ternary function W~i = f(Wi|Δ) to map weights to {-1,0,1}.
• Derive α* and Δ* to solve the reduced optimization, with Δ* approximated using a rule-of-thumb based on W statistics (Δ* ≈ 0.75 E|W|).
• Train networks by keeping full-precision copies of weights for updates while applying ternarization in forward/backward passes.
• Use SGD with momentum, batch normalization, and learning-rate step decay, following a procedure that updates W and α across layers.
• Infer by applying the scaling factor α to the ternary weights, enabling deployment with reduced storage.

Experimental results

Research questions

• RQ1 Can ternary weights (+1, 0, -1) with a scaling factor approximate full-precision weights effectively while reducing memory and computation?
• RQ2 How to optimally select the ternarization threshold Δ to balance accuracy and compression?
• RQ3 Do TWNs offer competitive performance compared with full-precision networks and binary-weight networks on classification and detection tasks?
• RQ4 What is the impact of TWNs on model size and computation relative to FPWNs and BPWNs across standard vision benchmarks?
• RQ5 Is the proposed training strategy (forward/backward ternarization with full-precision weight updates) practical and stable across architectures?

Key findings

• TWNs achieve up to 16x model compression with maintained convolutional operations similar to binary nets.
• On MNIST and CIFAR-10 TWNs near FPWNs in accuracy and outperform BPWNs.
• On ImageNet, TWNs bridge much of the gap to FPWNs, with smaller accuracy loss than BPWNs, especially with larger backbones.
• In object detection (PASCAL VOC with YOLOv5), TWNs outperform BPWNs by more than 10% mAP, approaching FPWNs.
• TWNs converge as fast and stably as FPWNs, outperforming BPWNs in training stability.