[Paper Review] Neural networks on microcontrollers: saving memory at inference via operator reordering

Edgar Liberis, Nicholas D. Lane|arXiv (Cornell University)|Oct 2, 2019

Advanced Memory and Neural ComputingEngineering20 citations

TL;DR

This paper proposes a software-level optimization to reduce peak memory usage in neural network inference on microcontrollers by reordering operator execution, enabling deployment of models that would otherwise exceed SRAM limits. By dynamically managing tensor buffer allocation and defragmenting memory after each operator, the method reduces peak memory by 50KB (from 351KB to 301KB), allowing a 250KB model to fit in 512KB SRAM without altering the model architecture or weights.

ABSTRACT

Designing deep learning models for highly-constrained hardware would allow imbuing many edge devices with intelligence. Microcontrollers (MCUs) are an attractive platform for building smart devices due to their low cost, wide availability, and modest power usage. However, they lack the computational resources to run neural networks as straightforwardly as mobile or server platforms, which necessitates changes to the network architecture and the inference software. In this work, we discuss the deployment and memory concerns of neural networks on MCUs and present a way of saving memory by changing the execution order of the network's operators, which is orthogonal to other compression methods. We publish a tool for reordering operators of TensorFlow Lite models and demonstrate its utility by sufficiently reducing the memory footprint of a CNN to deploy it on an MCU with 512KB SRAM.

Motivation & Objective

• Address the challenge of deploying deep learning models on microcontrollers (MCUs) with limited on-chip SRAM and no intermediate memory hierarchy.
• Overcome the constraint that MCU inference software must support dynamic memory allocation due to insufficient static memory for all tensor buffers.
• Minimize peak memory usage during inference without modifying model architecture or weights, enabling deployment of previously infeasible models.
• Develop a practical tool for reordering operators in TensorFlow Lite models to reduce memory footprint on resource-constrained hardware.

Proposed method

• Reorder the execution of operators in a neural network's computation graph to minimize the peak working set size (i.e., maximum number of tensors in memory at once).
• Implement a dynamic memory allocator for the TensorFlow Lite micro-interpreter that allows defragmentation by moving tensor buffers to the start of memory after each operator execution.
• Use a topological ordering strategy that prioritizes operators with minimal memory footprint and avoids holding intermediate tensors longer than necessary.
• Apply the reordering algorithm to a computation graph with branches (e.g., residual connections) to reduce memory pressure in complex architectures.
• Ensure that C/C++ pointers to tensor buffers are not retained across relocations, enabling safe memory movement during defragmentation.
• Integrate the reordering logic into a tool for TensorFlow Lite models, making it transparent to model developers and orthogonal to other compression techniques.

Experimental results

Research questions

• RQ1 Can reordering the execution order of operators in a neural network reduce peak memory usage on microcontrollers without changing the model architecture?
• RQ2 How much memory reduction can be achieved through operator reordering in models with complex, branched computation graphs?
• RQ3 What is the performance overhead of dynamic memory allocation and defragmentation in a microcontroller environment?
• RQ4 Can operator reordering make models with large peak memory usage deployable on MCUs with limited SRAM (e.g., 512KB) that would otherwise be infeasible?
• RQ5 How does the proposed method compare to static memory allocation in terms of memory efficiency and runtime cost on MCU platforms?

Key findings

• Operator reordering reduced the peak memory usage of SwiftNet Cell from 351KB to 301KB, a 50KB (14.2%) reduction, enabling deployment on a 512KB SRAM MCU.
• The dynamic memory allocator introduced only a 0.68% increase in execution time (10243ms → 1316ms) and 0.97% increase in energy use (8775mJ → 728mJ) for MobileNet-v1, with a 186KB reduction in memory footprint.
• The method is fully orthogonal to other compression techniques such as quantization, pruning, and knowledge distillation, allowing it to be combined with them.
• Defragmentation after each operator execution was effective and incurred minimal runtime overhead, making dynamic memory management viable on MCUs.
• The tool successfully enabled deployment of a 250KB model on a microcontroller with only 512KB SRAM, which was not possible with the default operator order.
• The approach is generalizable and can be extended to support optimizations like in-place computation (e.g., reusing input buffers for outputs) to further reduce memory usage.