[Paper Review] MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers

Colby Banbury, Chuteng Zhou+7 more|arXiv (Cornell University)|Oct 21, 2020

Advanced Neural Network ApplicationsComputer Science51 references148 citations

TL;DR

MicroNets use differentiable neural architecture search (DNAS) to design MCU-optimized networks that fit TinyML constraints, achieving state-of-the-art results for VWW, KWS, and AD with TensorFlow Lite Micro on common MCUs.

ABSTRACT

Executing machine learning workloads locally on resource constrained microcontrollers (MCUs) promises to drastically expand the application space of IoT. However, so-called TinyML presents severe technical challenges, as deep neural network inference demands a large compute and memory budget. To address this challenge, neural architecture search (NAS) promises to help design accurate ML models that meet the tight MCU memory, latency and energy constraints. A key component of NAS algorithms is their latency/energy model, i.e., the mapping from a given neural network architecture to its inference latency/energy on an MCU. In this paper, we observe an intriguing property of NAS search spaces for MCU model design: on average, model latency varies linearly with model operation (op) count under a uniform prior over models in the search space. Exploiting this insight, we employ differentiable NAS (DNAS) to search for models with low memory usage and low op count, where op count is treated as a viable proxy to latency. Experimental results validate our methodology, yielding our MicroNet models, which we deploy on MCUs using Tensorflow Lite Micro, a standard open-source NN inference runtime widely used in the TinyML community. MicroNets demonstrate state-of-the-art results for all three TinyMLperf industry-standard benchmark tasks: visual wake words, audio keyword spotting, and anomaly detection. Models and training scripts can be found at github.com/ARM-software/ML-zoo.

Motivation & Objective

• Demonstrate that operation count is a viable proxy for MCU model latency and energy under a uniform model space prior.
• Show that differentiable NAS with MCU-aware constraints can yield memory- and latency-efficient models.
• Provide state-of-the-art MicroNets for Visual Wake Words, Keyword Spotting, and Anomaly Detection within the TinyMLperf framework.

Proposed method

• Characterize MCU inference performance to establish op count as a latency proxy.
• Formulate a differentiable NAS (DNAS) objective with memory (eFlash, SRAM) and latency constraints and sub-byte quantization options.
• Define MCU-specific backbones for VWW, KWS, and AD as search spaces and optimize via DNAS with memory/latency regularization.
• Incorporate 4-bit quantization emulation within CMSIS-NN/TFLM to expand the search space under hardware constraints.
• Train discovered architectures with quantization-aware training and knowledge distillation where applicable.
• Deploy final models via TensorFlow Lite Micro and evaluate on standard TinyMLperf tasks.

Experimental results

Research questions

• RQ1 Can MCU latency and energy be effectively approximated by the number of operations (ops) for end-to-end models within a given backbone?
• RQ2 Can DNAS be constrained to meet MCU SRAM/eFlash and latency limits while maximizing accuracy?
• RQ3 Do MCU-optimized MicroNets achieve state-of-the-art accuracy and throughput on TinyMLperf tasks VWW, KWS, and AD when deployed with TFLM?

Key findings

• Ops is a viable proxy for end-to-end model latency on MCUs within a backbone, despite per-layer variability.
• MCU power is largely independent of model size, making energy per inference primarily a function of MCU size and model ops.
• DNAS with MCU-aware constraints can produce architectures that fit eFlash and SRAM while maintaining high accuracy and acceptable latency.
• MicroNets achieve Pareto-optimal trade-offs for small and medium MCUs on VWW and KWS tasks.
• On VWW, the MicroNet for the medium MCU achieves 88.03% accuracy, closely matching MobileNetV2 88.75% while enabling deployment on the target MCU; for the small MCU, MicroNet is 3.1% more accurate than the TFLM reference and 21 ms faster.
• For KWS, the MicroNet medium model is 2.7× faster than DS-CNN(L) and more accurate.