[Paper Review] Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search

BoRui Wu, Yanghan Wang+4 more|arXiv (Cornell University)|Nov 30, 2018

Advanced Neural Network Applications24 references199 citations

TL;DR

This paper formulates mixed precision quantization as a neural architecture search problem and introduces differentiable neural architecture search (DNAS) to assign layer-wise bit-widths, achieving substantial compression with maintained accuracy on ResNet models. The approach is notably faster than traditional NAS and adaptable to various hardware cost metrics.

ABSTRACT

Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources. However, existing quantization methods often represent all weights and activations with the same precision (bit-width). In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths. We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization. Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet. Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

Motivation & Objective

• Motivate reducing inference cost and memory footprint by allowing different bit-widths per layer rather than uniform quantization.
• Propose an efficient NAS-based framework (DNAS) to search over layer-wise precisions.
• Demonstrate that mixed-precision quantization can surpass state-of-the-art compression while maintaining or improving accuracy.
• Show that the DNAS approach is fast and scalable for large networks like ResNet on ImageNet.

Proposed method

• Represent the architecture space as a stochastic super net where edges are convolution operators with different quantization bit-widths.
• Relax the discrete edge choices with architecture parameters and use Gumbel SoftMax to enable differentiable gradient-based optimization.
• Train weights and architecture parameters jointly with SGD, sampling candidate architectures from the learned distribution.
• Define a cost-aware objective that balances accuracy (cross-entropy) with model size or compute cost via a configurable Cost(a) function.
• Apply the DNAS framework to ResNet on CIFAR-10 and ImageNet to find layer-wise precision assignments.
• Quantize weights and activations following DoReFa-Net and PACT, with block-level mixed-precision search and temperature-annealed Gumbel sampling.

Experimental results

Research questions

• RQ1 Can mixed-precision quantization, optimized via differentiable NAS, outperform uniform-precision quantization in terms of accuracy and compression?
• RQ2 How can one efficiently search exponential bit-width configurations across layers without prohibitive compute?
• RQ3 What is the impact of layer-wise bit-width choices on model size and FLOP reduction for large architectures like ResNet on ImageNet?
• RQ4 Is the DNAS framework fast enough to be practical for real-world model compression tasks?

Key findings

• Quantized models achieve up to 21.1x model size reduction or 103.9x lower computational cost while outperforming baseline quantized or even full-precision models in some cases.
• On CIFAR-10, mixed-precision ResNet variants surpass full-precision accuracy by up to 0.37% with 11.6–16.6x compression; some configurations reach 16.6–20.3x compression with sub-0.39% accuracy loss.
• On ImageNet, best architectures for ResNet-18/34 achieve 0.18–0.49% accuracy gains with about 10.6–11.2x model size reduction (MA) or reach 19.0–21.1x reductions with competitive accuracy (ME).
• Compared to TTQ and ADMM baselines, DNAS architectures maintain higher accuracy at similar or greater compression levels for both model size and computational cost experiments.
• The DNAS pipeline completes a search on ResNet-18 for ImageNet in under 5 hours on 8 V100 GPUs, demonstrating practical efficiency over prior NAS approaches.