[Paper Review] TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

Robert David, Jared Duke+11 more|arXiv (Cornell University)|Oct 17, 2020

Advanced Neural Network ApplicationsComputer Science17 references167 citations

TL;DR

The paper presents TensorFlow Lite Micro (TFLM), an interpreter-based, portable ML inference framework designed for embedded TinyML devices, enabling cross-platform deployment with minimal runtime overhead and memory footprint.

ABSTRACT

Deep learning inference on embedded devices is a burgeoning field with myriad applications because tiny embedded devices are omnipresent. But we must overcome major challenges before we can benefit from this opportunity. Embedded processors are severely resource constrained. Their nearest mobile counterparts exhibit at least a 100 -- 1,000x difference in compute capability, memory availability, and power consumption. As a result, the machine-learning (ML) models and associated ML inference framework must not only execute efficiently but also operate in a few kilobytes of memory. Also, the embedded devices' ecosystem is heavily fragmented. To maximize efficiency, system vendors often omit many features that commonly appear in mainstream systems, including dynamic memory allocation and virtual memory, that allow for cross-platform interoperability. The hardware comes in many flavors (e.g., instruction-set architecture and FPU support, or lack thereof). We introduce TensorFlow Lite Micro (TF Micro), an open-source ML inference framework for running deep-learning models on embedded systems. TF Micro tackles the efficiency requirements imposed by embedded-system resource constraints and the fragmentation challenges that make cross-platform interoperability nearly impossible. The framework adopts a unique interpreter-based approach that provides flexibility while overcoming these challenges. This paper explains the design decisions behind TF Micro and describes its implementation details. Also, we present an evaluation to demonstrate its low resource requirement and minimal run-time performance overhead.

Motivation & Objective

• Identify challenges in deploying ML on fragmented embedded hardware and tight resource limits.
• Propose a portable, interpreter-based ML inference framework for microcontrollers and similar devices.
• Demonstrate design decisions that enable low memory use, portability, and vendor kernel optimizations.
• Show how to leverage TensorFlow Lite tooling to export and run models on embedded targets.

Proposed method

• Adopt an interpreter-based inference approach to maximize portability and reduce re-exporting models across devices.
• Reuse TensorFlow Lite model format and FlatBuffer serialization to load models without unpacking.
• Implement a two-stack memory arena and a memory planner to minimize runtime and persistent memory.
• Support multitenancy by sharing a single arena across multiple interpreters.
• Enable platform specialization by swapping in vendor-optimized kernels (e.g., CMSIS-NN) without altering build scripts.
• Provide a platform-agnostic build system to span heterogeneous embedded toolchains.

Experimental results

Research questions

• RQ1 Can an interpreter-based ML inference framework meet the resource constraints of embedded TinyML devices while remaining portable across hardware platforms?
• RQ2 How can memory management and memory planning be designed to minimize arena footprint for repeated inferences on microcontrollers?
• RQ3 To what extent can vendor-optimized kernels be integrated without sacrificing portability and maintainability?
• RQ4 How effectively can existing TensorFlow Lite tooling be reused for model export and deployment on embedded targets?

Key findings

• TFLM demonstrates low resource requirements and minimal run-time overhead for embedded inference.
• An interpreter-based approach can be suitable for embedded ML due to amortized kernel complexity.
• The reuse of TensorFlow Lite tools enables easy export of models to embedded targets.
• A two-stack memory allocation strategy and a memory-planner reduce arena size and enable memory reuse.
• Platform specialization via kernel swapping (e.g., CMSIS-NN) achieves performance gains without changing build systems.