[Paper Review] Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

Urs Köster, Tristan J. Webb+12 more|arXiv (Cornell University)|Nov 6, 2017

Model Reduction and Neural Networks11 references166 citations

TL;DR

Flexpoint is a 16-bit mantissa with a 5-bit shared exponent format for training deep nets, managed by Autoflex, achieving parity with 32-bit FP training on AlexNet, ResNet, and WGAN without hyperparameter changes.

ABSTRACT

Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neural networks in low bit-width remains a challenging problem. Here we present the Flexpoint data format, aiming at a complete replacement of 32-bit floating point format training and inference, designed to support modern deep network topologies without modifications. Flexpoint tensors have a shared exponent that is dynamically adjusted to minimize overflows and maximize available dynamic range. We validate Flexpoint by training AlexNet, a deep residual network and a generative adversarial network, using a simulator implemented with the neon deep learning framework. We demonstrate that 16-bit Flexpoint closely matches 32-bit floating point in training all three models, without any need for tuning of model hyperparameters. Our results suggest Flexpoint as a promising numerical format for future hardware for training and inference.

Motivation & Objective

• Demonstrate that a shared-exponent fixed-point style format can train modern neural networks without topology or hyperparameter changes.
• Develop Autoflex to dynamically manage tensor exponents and prevent overflows during training.
• Evaluate Flexpoint on representative architectures to compare training performance with 32-bit floating point.

Proposed method

• Propose flexN+M data format where tensors share a common exponent and maintain a fixed mantissa.
• Introduce Autoflex to predict and adjust output exponents per operation based on tensor statistics.
• Use a initialization procedure to set exponents and a two-stage per-minibatch scaling update during training.
• Validate via simulator experiments that map Flexpoint operations to GPU kernels using int16 with float32 computation under the hood.
• Compare training dynamics and final performance against FP32 and FP16 baselines across models.

Experimental results

Research questions

• RQ1 Can a shared-exponent, fixed-point-like format achieve parity with FP32 training across diverse networks without altering hyperparameters?
• RQ2 How can an exponent management strategy (Autoflex) prevent overflows and efficiently utilize dynamic range during training?
• RQ3 Do fixed-point-inspired formats degrade training quality for challenging models like ResNet or GANs compared to FP32?
• RQ4 What is the empirical impact of Flexpoint on convergence and final accuracy for CNNs and GANs?

Key findings

• Flexpoint with flex16+5 closely matches FP32 training performance for AlexNet, ResNet-110 on CIFAR-10, and Wasserstein GAN on LSUN.
• FP16 without adjustments underperforms compared with FP32 and Flexpoint for these models.
• Autoflex can predict and adjust exponents to prevent overflows, enabling stable training without hyperparameter changes.
• Experiments use a GPU-based Flexpoint simulator showing comparable training dynamics to FP32 and better stability than FP16.
• Flexpoint offers hardware efficiency advantages by converting many operations to fixed-point style arithmetic while preserving algorithmic parity with FP32.