[Paper Review] Mixed Precision Training of Convolutional Neural Networks using Integer Operations

Dipankar Das, Naveen Mellempudi+15 more|arXiv (Cornell University)|Feb 3, 2018

Advanced Neural Network Applications11 references114 citations

TL;DR

The paper demonstrates INT16-based mixed-precision CNN training on ImageNet-1K, achieving or surpassing FP32 accuracy with about 1.8x end-to-end training throughput on GP hardware, using a Dynamic Fixed Point scheme and integer FMA kernels.

ABSTRACT

The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half-precision

Motivation & Objective

• Demonstrate training of state-of-the-art CNNs on ImageNet-1K using INT16 integer operations on general-purpose hardware.
• Develop a Dynamic Fixed Point (DFP) representation with a shared tensor-wide exponent for neural network training.
• Design efficient integer-based kernels for convolution and GEMM; manage overflow and scaling across layers.
• Show that mixed-precision INT16 training matches or exceeds FP32 accuracy without hyper-parameter changes and with throughput gains.

Proposed method

• Propose a Dynamic Fixed Point (DFP) format with a shared exponent per tensor and INT16 data elements.
• Use INT16 tensors with a shared 8-bit signed exponent to implement GEMM/conv operations via integer FMA instructions.
• Develop down-conversion from DFP-32 to DFP-16 using max-value-based scaling and multiple rounding strategies.
• Employ partial accumulation in INT32 with occasional FP32 accumulation to prevent overflows during FMA chains.
• Retain FP32 master copies of weights while updating with FP32 gradients; quantize activations/weights for next layer using DFP-16 primitives.

Experimental results

Research questions

• RQ1 Can INT16-based mixed-precision training on general-purpose hardware match or surpass FP32 accuracy on large CNNs trained from scratch on ImageNet-1K?
• RQ2 What is the impact of Dynamic Fixed Point representation and shared exponents on accuracy and performance in CNN training?
• RQ3 What is the potential speedup and hardware efficiency when using integer FMA kernels (e.g., AVX512_4VNNI) for forward/backward passes and weight updates?
• RQ4 How to manage overflow and scaling across convolution and GEMM kernels to maintain numerical stability during training?

Key findings

• DFP16 mixed-precision training matches or exceeds FP32 accuracy for ResNet-50, GoogLeNet-v1, VGG-16, and AlexNet on ImageNet-1K.
• ResNet-50 achieved 75.77% Top-1 and 92.84% Top-5 with DFP16, surpassing the FP32 baseline of 75.70% Top-1 and 92.78% Top-5.
• Average end-to-end training throughput improves by 1.8x over FP32 on the tested hardware.
• Convolution kernels: 3x3 are 1.8x faster and 1x1 are 1.4x faster; batchnorm 2x faster; ReLU/EltWise fused with BN, enhancing memory bandwidth efficiency.
• GoogLeNet-v1, VGG-16, and AlexNet also reach state-of-the-art accuracy with INT16 training, without hyper-parameter changes.
• Demonstrates first INT16 CNN training results on GP hardware for ImageNet-1K with SOTA CNNs and highest reported accuracy with half-precision representation.