[Paper Review] MCUNet: Tiny Deep Learning on IoT Devices

Ji Lin, Wei-Ming Chen+4 more|arXiv (Cornell University)|Jul 20, 2020

Advanced Neural Network Applications51 references255 citations

TL;DR

MCUNet jointly designs a tiny neural network (TinyNAS) and a memory-efficient inference engine (TinyEngine) to enable ImageNet-scale deep learning on off-the-shelf microcontrollers, achieving 70.7% top-1 and fast wake-word performance within tight memory budgets.

ABSTRACT

Machine learning on tiny IoT devices based on microcontroller units (MCU) is appealing but challenging: the memory of microcontrollers is 2-3 orders of magnitude smaller even than mobile phones. We propose MCUNet, a framework that jointly designs the efficient neural architecture (TinyNAS) and the lightweight inference engine (TinyEngine), enabling ImageNet-scale inference on microcontrollers. TinyNAS adopts a two-stage neural architecture search approach that first optimizes the search space to fit the resource constraints, then specializes the network architecture in the optimized search space. TinyNAS can automatically handle diverse constraints (i.e.device, latency, energy, memory) under low search costs.TinyNAS is co-designed with TinyEngine, a memory-efficient inference library to expand the search space and fit a larger model. TinyEngine adapts the memory scheduling according to the overall network topology rather than layer-wise optimization, reducing the memory usage by 4.8x, and accelerating the inference by 1.7-3.3x compared to TF-Lite Micro and CMSIS-NN. MCUNet is the first to achieves >70% ImageNet top1 accuracy on an off-the-shelf commercial microcontroller, using 3.5x less SRAM and 5.7x less Flash compared to quantized MobileNetV2 and ResNet-18. On visual&audio wake words tasks, MCUNet achieves state-of-the-art accuracy and runs 2.4-3.4x faster than MobileNetV2 and ProxylessNAS-based solutions with 3.7-4.1x smaller peak SRAM. Our study suggests that the era of always-on tiny machine learning on IoT devices has arrived. Code and models can be found here: https://tinyml.mit.edu.

Motivation & Objective

• Motivate and enable ImageNet-scale deep learning on microcontrollers with extremely limited SRAM/Flash.
• Develop a system-algorithm co-design framework combining neural architecture search and inference scheduling to minimize peak memory and maximize accuracy.
• Automate the search space optimization to fit diverse tiny hardware constraints.
• Provide a memory-efficient inference library that expands the feasible model space for tiny devices.

Proposed method

• TinyNAS performs a two-stage NAS: automatic search-space optimization by analyzing FLOPs distributions of satisfying networks across 108 search-space configurations, then one-shot NAS with weight sharing and evolution search within the chosen space.
• TinyEngine uses code-generation to eliminate runtime overhead, model-adaptive memory scheduling, kernel specialization, and in-place depth-wise convolution to reduce peak memory and boost throughput.
• The framework co-designs TinyNAS with TinyEngine to expand the feasible model capacity under MCU memory budgets.
• Quantization to int8 is used to deploy models, with exploration of 4-bit quantization to fit larger models under memory limits.
• Evaluation is performed on ImageNet, Visual Wake Words, and Speech Commands across multiple MCUs (e.g., STM32F746, F412, H743).

Experimental results

Research questions

• RQ1 Can system-level co-design of neural architecture and inference runtime enable ImageNet-scale models on memory-constrained MCUs?
• RQ2 How much memory and latency can be saved by co-designing architecture search with a memory-aware inference engine?
• RQ3 What are the effects of search-space optimization and memory scheduling on final accuracy under tight SRAM/Flash budgets?
• RQ4 Is int8 (and lower bit) quantization sufficient to achieve competitive accuracy on MCU-bound models?
• RQ5 How does MCUNet perform on wake-word and object-detection tasks compared to existing TinyML baselines?

Key findings

• MCUNet achieves a record ImageNet top-1 accuracy of 70.7% on off-the-shelf microcontrollers.
• TinyEngine reduces peak memory by 3.4× and accelerates inference by 1.7–3.3× versus TF-Lite Micro and CMSIS-NN.
• With TinyEngine and TinyNAS, MobileNetV2 and ProxylessNAS baselines improve to 61.8% top-1 accuracy under tight memory, versus 47.4–56.4% with library alone.
• MCUNet runs wake-word tasks 2.4–3.4× faster with 3.7–4.1× smaller peak SRAM compared to baselines on wake-word datasets (VWW and Speech Commands).
• Object detection on Pascal VOC under 512kB SRAM shows MCUNet mAP of 51.4% versus 31.6% for MobileNetV2+CMSIS-NN under memory constraints.
• MCUNet reduces memory (SRAM) by about 3.5× and Flash by about 5.7× while achieving higher ImageNet accuracy, compared to ResNet-18/MobileNetV2 equivalents in 8-bit.