[Paper Review] Mixed Precision DNNs: All you need is a good parametrization

Stefan Uhlich, Lukas Mauch+6 more|arXiv (Cornell University)|May 27, 2019

Advanced Neural Network ApplicationsComputer Science22 references51 citations

TL;DR

The paper shows that parametrizing the quantizer by its step size and dynamic range (which yields inferred bitwidths) enables stable training and state-of-the-art mixed-precision DNNs.

ABSTRACT

Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.

Motivation & Objective

• Motivate efficient DNN inference on mobile/embedded devices through mixed precision quantization.
• Investigate how to learn quantization parameters end-to-end using gradient-based methods.
• Identify a parametrization that enables stable training and strong final performance.
• Demonstrate the impact of learned quantization on CIFAR-10 and ImageNet comparisons.

Proposed method

• Use differentiable quantization with straight-through gradients to learn quantizer parameters.
• Parametrize the quantizer by step size and dynamic range, from which bitwidth is inferred.
• Compare parametrizations that do and do not explicitly use bitwidth.
• Evaluate on CIFAR-10 and ImageNet to assess performance of learned mixed-precision networks.

Experimental results

Research questions

• RQ1 Can differentiable quantization learn useful bitwidths when parameterized by step size and dynamic range?
• RQ2 Does this parametrization yield more stable training and better performance than bitwidth-based parametrizations?
• RQ3 How do learned quantization parameters affect performance on standard benchmarks (CIFAR-10, ImageNet) compared to fixed or homogeneous precision?
• RQ4 What is the impact of learned mixed-precision networks on inference efficiency under size constraints?

Key findings

• Parametrizing the quantizer by step size and dynamic range enables stable training of mixed-precision DNNs.
• Bitwidth can be inferred from the learned step size and dynamic range rather than being fixed a priori.
• Other parametrizations that explicitly use bitwidth perform worse.
• Experiments on CIFAR-10 and ImageNet show learned quantization achieves strong performance.
• The approach yields mixed-precision DNNs with learned quantization parameters that achieve state-of-the-art performance on the tested benchmarks.