[Paper Review] Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers

Manuele Rusci, Alessandro Capotondi+1 more|arXiv (Cornell University)|May 30, 2019

Advanced Memory and Neural ComputingEngineering41 citations

TL;DR

The paper proposes a memory-driven mixed-precision quantization method and an Integer Channel-Normalization (ICN) activation to enable end-to-end integer-only deep network inference on microcontrollers, achieving 68% Top1 on a STM32H7 with 2MB Flash and 512kB RAM, 8% higher than prior 8-bit methods.

ABSTRACT

This paper presents a novel end-to-end methodology for enabling the deployment of low-error deep networks on microcontrollers. To fit the memory and computational limitations of resource-constrained edge-devices, we exploit mixed low-bitwidth compression, featuring 8, 4 or 2-bit uniform quantization, and we model the inference graph with integer-only operations. Our approach aims at determining the minimum bit precision of every activation and weight tensor given the memory constraints of a device. This is achieved through a rule-based iterative procedure, which cuts the number of bits of the most memory-demanding layers, aiming at meeting the memory constraints. After a quantization-aware retraining step, the fake-quantized graph is converted into an inference integer-only model by inserting the Integer Channel-Normalization (ICN) layers, which introduce a negligible loss as demonstrated on INT4 MobilenetV1 models. We report the latency-accuracy evaluation of mixed-precision MobilenetV1 family networks on a STM32H7 microcontroller. Our experimental results demonstrate an end-to-end deployment of an integer-only Mobilenet network with Top1 accuracy of 68% on a device with only 2MB of FLASH memory and 512kB of RAM, improving by 8% the Top1 accuracy with respect to previously published 8 bit implementations for microcontrollers.

Motivation & Objective

• Automate the selection of minimum bit precision for each activation and weight tensor under device memory constraints.
• Enable end-to-end integer-only inference on microcontrollers through mixed-precision quantization.
• Introduce the Integer Channel-Normalization (ICN) activation to support sub-byte, per-channel quantization.
• Demonstrate latency-accuracy tradeoffs for MobilenetV1 family on a STM32H7 MCU under strict memory budgets.

Proposed method

• Use mixed-precision quantization with 8, 4, or 2-bit uniform quantization for activations and weights.
• Model the inference with integer-only operations and a rule-based iterative procedure to satisfy memory constraints.
• Introduce and apply Integer Channel-Normalization (ICN) layers to convert fake-quantized graphs into integer-only deployments.
• Perform quantization-aware retraining to recover accuracy after quantization.
• Support per-channel quantization and avoid folding batch normalization parameters into weights to prevent accuracy collapse.
• Evaluate latency-accuracy on MobilenetV1 family running on STM32H7 with CMSIS-NN optimizations.

Experimental results

Research questions

• RQ1 Can memory-constrained microcontrollers support end-to-end integer-only deep networks through mixed-precision quantization?
• RQ2 How to automatically assign per-tensor bit-widths to meet device memory constraints while preserving accuracy?
• RQ3 What is the impact of Integer Channel-Normalization (ICN) on accuracy and deployment feasibility for sub-byte quantization?
• RQ4 How does per-channel ICN compare to per-layer quantization in terms of accuracy and memory footprint on MCU targets?

Key findings

• An end-to-end integer-only MobilenetV1 deployment on a STM32H7 (2MB Flash, 512kB RAM) achieves 68% Top1 accuracy.
• The 68% model is 8% higher than previously published 8-bit integer-only implementations for microcontrollers.
• PL+ICN INT4 prevents BN folding-induced accuracy collapse, enabling training convergence where PL+FB INT4 collapses.
• PC+ICN INT4 yields 66.41% Top1 accuracy with a memory footprint around 2.12 MB, demonstrating the benefit of per-channel quantization with ICN.
• MixQ-PC-ICN configurations provide Pareto-efficient accuracy-latency tradeoffs, with PC+ICN achieving higher accuracy at modest latency overhead (~20%) compared to MixQ-PL.
• On 2MB ROM / 512kB RAM budgets, some configurations achieve up to 68% Top1 accuracy, outperforming several prior mixed-precision approaches under similar constraints.