[Paper Review] Rethinking floating point for deep learning

Jeff Johnson|arXiv (Cornell University)|Nov 1, 2018

Neural Networks and ApplicationsComputer Science25 references104 citations

TL;DR

The paper introduces an 8-bit log float with Kulisch accumulation and ELMA, achieving near float32 accuracy on ResNet-50 without retraining, and showing energy/area advantages over 8/32-bit integer MACs on 28 nm ASIC hardware.

ABSTRACT

Reducing hardware overhead of neural networks for faster or lower power inference and training is an active area of research. Uniform quantization using integer multiply-add has been thoroughly investigated, which requires learning many quantization parameters, fine-tuning training or other prerequisites. Little effort is made to improve floating point relative to this baseline; it remains energy inefficient, and word size reduction yields drastic loss in needed dynamic range. We improve floating point to be more energy efficient than equivalent bit width integer hardware on a 28 nm ASIC process while retaining accuracy in 8 bits with a novel hybrid log multiply/linear add, Kulisch accumulation and tapered encodings from Gustafson's posit format. With no network retraining, and drop-in replacement of all math and float32 parameters via round-to-nearest-even only, this open-sourced 8-bit log float is within 0.9% top-1 and 0.2% top-5 accuracy of the original float32 ResNet-50 CNN model on ImageNet. Unlike int8 quantization, it is still a general purpose floating point arithmetic, interpretable out-of-the-box. Our 8/38-bit log float multiply-add is synthesized and power profiled at 28 nm at 0.96x the power and 1.12x the area of 8/32-bit integer multiply-add. In 16 bits, our log float multiply-add is 0.59x the power and 0.68x the area of IEEE 754 float16 fused multiply-add, maintaining the same signficand precision and dynamic range, proving useful for training ASICs as well.

Motivation & Objective

• Investigate whether floating-point representations can be made more energy-efficient for neural networks without retraining or extensive quantization.
• Develop a new arithmetic that preserves general floating-point semantics while improving dynamic range and energy efficiency at small word sizes.
• Evaluate the proposed arithmetic on standard CNNs and compare to traditional integer quantization baselines on ASIC and FPGA platforms.

Proposed method

• Propose an 8-bit log-domain representation with tapering inspired by posit (N,s) and Gustafson’s tapering, plus an ELMA (exact log-linear multiply-add) via Kulisch accumulation.
• Use a combination of log-domain multiplication and linear-domain accumulation to enable energy-efficient EMA/ELMA operations.
• Replace all math and float32 parameters with round-to-nearest-even conversions as a drop-in replacement, without network retraining.
• Synthesize and evaluate on 28 nm ASIC with a 32x32 matrix multiply-accumulate and a 8/38-bit log MADD, comparing power/area to IEEE/standard alternatives.

Experimental results

Research questions

• RQ1 Can 8-bit floating-point-like representations retain near float32 accuracy on CNNs without retraining?
• RQ2 Do log-domain and Kulisch accumulation techniques provide energy and area advantages over conventional int8/32 quantization on hardware?
• RQ3 What are the trade-offs in accuracy, latency, and hardware resources when adopting ELMA-based multiply-add for CNN workloads?

Key findings

• 8-bit log float with ELMA achieves top-1 accuracy within 0.9% and top-5 accuracy within 0.2% of float32 ResNet-50 on ImageNet without retraining.
• On 28 nm ASIC, 8/38-bit log MADD consumes 0.96x the power and 1.12x the area of 8/32-bit integer MADD.
• In 16-bit configuration, log MADD uses 0.59x the power and 0.68x the area of IEEE-754 float16 FMA while maintaining comparable significand precision and dynamic range.
• FPGA experiments show competitive results against int8/32 MAC baselines using ELMA with Kulisch accumulation.