[Paper Review] Convolutional Neural Networks using Logarithmic Data Representation

Daisuke Miyashita, Edward H. Lee+1 more|arXiv (Cornell University)|Mar 3, 2016

Advanced Neural Network Applications25 references362 citations

TL;DR

This paper introduces log-domain representations to quantize weights and activations to as few as 3–4 bits, enabling near-floating-point accuracy with reduced multipliers, and demonstrates end-to-end training with log-based quantization.

ABSTRACT

Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.

Motivation & Objective

• Motivate low-precision CNNs by exploiting non-uniform weight/activation distributions.
• Propose log-domain representations for activations, weights, and dot-products to reduce hardware complexity.
• Demonstrate improved or comparable accuracy with 3–4 bit quantization against fixed-point baselines.
• Explore different log bases (e.g., base-2, base-√2) to balance dynamic range and precision.
• Present an end-to-end training procedure using log-domain representations and quantized gradients.

Proposed method

• Represent dot products in the log-domain to replace multipliers with bit-shifts (two variants: single-operand log and dual-operand log).
• Quantize activations and weights in log2 space via LogQuant layers; compare with linear quantization for equivalent bitwidths.
• Use accumulation strategies in log-domain leveraging log(1+x)≈x for small x to approximate sums.
• Generalize arithmetic to different bases (base-2 and base-√2) to capture wide dynamic ranges with finer precision.
• Develop an end-to-end training approach with log-domain representations including gradients (Algorithm 1).
• Evaluate on AlexNet and VGG16 for ILSVRC-2012 and on CIFAR-10 with a 5-bit log-network.

Experimental results

Research questions

• RQ1 Can 3-bit log-domain representations for activations and weights achieve comparable top-5 accuracy to 32-bit floating-point nets on ImageNet models?
• RQ2 How does log-domain quantization compare to linear quantization in terms of accuracy and robustness across AlexNet and VGG16?
• RQ3 Does a base-√2 logarithmic representation offer practical advantages over base-2 for capturing large dynamic ranges with finer granularity?
• RQ4 Is end-to-end training feasible with log-domain quantization, including quantized gradients, without substantial loss in performance?
• RQ5 What are the trade-offs in network size and accuracy when quantizing convolutional vs fully connected weights in the log domain?

Key findings

• 3-bit log activations achieve near FP performance on AlexNet and VGG16, with small losses compared to 32-bit float.
• For AlexNet, 3-bit log activations yield 76.9% top-5 (fsr=7) vs 78.3% FP32; 3-bit linear yields 77.1% (fsr=5).
• For VGG16, 3-bit log activations yield 89.2% top-5 (fsr=6) vs 89.8% FP32; 3-bit linear yields 83.0% (fsr=3).
• 4-bit log activations recover FP32 performance on VGG16 (89.8% top-5, fsr=11) and match AlexNet better than 4-bit linear.
• Base-2 vs base-√2 for 5-bit weight quantization in conv layers shows base-√2 markedly improves accuracy (AlexNet: 70.6% base-2 vs 75.1% base-√2; VGG16: 83.4% base-2 vs 89.0% base-√2).
• Training with log-domain representations (5-bit activations/weights/gradients) on CIFAR-10 achieves 0.9379 test accuracy for log vs 0.9253 for linear and 0.8862 for BinaryNet.