[Paper Review] Measuring what Really Matters: Optimizing Neural Networks for TinyML

Lennart Heim, Andreas Biri+2 more|arXiv (Cornell University)|Apr 21, 2021

Advanced Numerical Analysis Techniques26 citations

TL;DR

The paper demonstrates that perceptible metrics (latency, energy) on MCUs are not reliably predicted by proxies like MACC/ FLOPs and introduces an implementation-aware toolchain to benchmark and optimize NNs directly on ARM Cortex‑M devices for TinyML applications.

ABSTRACT

With the surge of inexpensive computational and memory resources, neural networks (NNs) have experienced an unprecedented growth in architectural and computational complexity. Introducing NNs to resource-constrained devices enables cost-efficient deployments, widespread availability, and the preservation of sensitive data. This work addresses the challenges of bringing Machine Learning to MCUs, where we focus on the ubiquitous ARM Cortex-M architecture. The detailed effects and trade-offs that optimization methods, software frameworks, and MCU hardware architecture have on key performance metrics such as inference latency and energy consumption have not been previously studied in depth for state-of-the-art frameworks such as TensorFlow Lite Micro. We find that empirical investigations which measure the perceptible metrics - performance as experienced by the user - are indispensable, as the impact of specialized instructions and layer types can be subtle. To this end, we propose an implementation-aware design as a cost-effective method for verification and benchmarking. Employing our developed toolchain, we demonstrate how existing NN deployments on resource-constrained devices can be improved by systematically optimizing NNs to their targeted application scenario.

Motivation & Objective

• Motivate the need for edge-enabled neural networks on memory- and energy-constrained microcontrollers.
• Investigate how optimization methods, software frameworks, and MCU hardware architecture affect perceptible metrics like latency and energy.
• Develop a complete hardware/software toolchain to measure and optimize neural networks at layer granularity on MCUs.
• Provide design guidelines for implementing hardware-aware TinyML by correlating perceptible metrics with architectural features.

Proposed method

• Develop an implementation-aware benchmarking toolchain that deploys NNs on MCUs to measure layer-level latency and energy.
• Quantify the relationship between perceptible metrics and optimization techniques (quantization, CMSIS-NN) on ARM Cortex-M MCUs.
• Compare on-host (TensorFlow/TFLite) versus on-device execution to assess accuracy loss and memory footprint.
• Use latency per operation delta = t_m / c_e to compare layer efficiency across architectures and hyper-parameters.
• Analyze how layer type, kernel parameters, and hardware features influence throughput and energy consumption.
• Demonstrate how hardware-aware NN design can improve accuracy within perceptible metric constraints.

Experimental results

Research questions

• RQ1 How well do common proxy metrics (e.g., number of operations) correlate with actual on-device latency and energy on Cortex-M MCUs?
• RQ2 What is the impact of 8-bit quantization and CMSIS-NN on inference latency and energy across different network architectures?
• RQ3 How do layer types and hyper-parameters interact with MCU architectural features to affect perceptible metrics?
• RQ4 Can an implementation-aware toolchain reliably guide NAS and NN design for TinyML deployments?
• RQ5 Which hardware targets (MCU models) offer Pareto-efficient trade-offs between latency and energy for given networks?

Key findings

• Quantization reduces memory footprint up to 73% with only 0.05% accuracy loss in some cases.
• On-device CMSIS-NN acceleration with 8-bit quantization yields up to ~4x speedup beyond an FPU-enabled float0n model, depending on the layer.
• Optimizations yield non-uniform acceleration across layers and architectures; larger networks (e.g., ResNet) can see 29x–35x speedups, while smaller networks see 12.9x–16x.
• Latency per operation varies by layer type and is not reliably predicted by total operation count; dense and convolution layers show significant gains from hardware-aware optimization, while depth-wise convolutions show smaller gains.
• Latency and energy consumption are almost perfectly linearly correlated (r ≈ 0.995) across optimizations and MCUs, but the best hardware choice depends on the Pareto trade-off for a given deployment.
• The choice of MCU can be Pareto-efficient (e.g., F4), and proxies alone are insufficient; empirical hardware-aware testing is essential.