[Paper Review] Not All Ops Are Created Equal!

Liangzhen Lai, Naveen Suda+1 more|arXiv (Cornell University)|Jan 12, 2018

Advanced Memory and Neural Computing12 references19 citations

TL;DR

This paper demonstrates that traditional neural network metrics—operations and parameters—fail to predict real-world efficiency on embedded devices. On an Arm Cortex-M7, operation types vary in throughput and energy use by up to 5X, and activation memory can contribute up to 30% of total memory footprint, necessitating holistic optimization beyond ops and parameters for efficient edge inference.

ABSTRACT

Efficient and compact neural network models are essential for enabling the deployment on mobile and embedded devices. In this work, we point out that typical design metrics for gauging the efficiency of neural network architectures -- total number of operations and parameters -- are not sufficient. These metrics may not accurately correlate with the actual deployment metrics such as energy and memory footprint. We show that throughput and energy varies by up to 5X across different neural network operation types on an off-the-shelf Arm Cortex-M7 microcontroller. Furthermore, we show that the memory required for activation data also need to be considered, apart from the model parameters, for network architecture exploration studies.

Motivation & Objective

• To challenge the assumption that operations and parameters alone are sufficient metrics for evaluating neural network efficiency on embedded systems.
• To quantify the variation in throughput and energy consumption across different neural network operation types on real hardware.
• To highlight the critical but often overlooked role of activation memory in total memory footprint for resource-constrained devices.
• To advocate for a holistic optimization framework that includes operation type, throughput, energy, and activation memory in neural architecture search.
• To provide empirical evidence from real measurements on an Arm Cortex-M7 for guiding efficient model deployment on microcontrollers.

Proposed method

• Measured throughput, power, and energy per operation across diverse neural network operation types (e.g., conv, FC, DS-Conv, max pooling) on a NUCLEO-F746ZG board with an Arm Cortex-M7.
• Used optimized CMSIS-NN kernels to ensure accurate and representative performance measurements on a general-purpose microcontroller.
• Analyzed the distribution of operation types across multiple depthwise separable CNN models trained on the Google Speech Commands dataset.
• Quantified maximum concurrent activation data size from network topology and execution order to estimate SRAM footprint.
• Correlated operation type distribution with energy consumption and throughput to identify performance bottlenecks.
• Pre-characterized operation performance per hardware platform to enable predictive estimation of network efficiency during architecture search.

Experimental results

Research questions

• RQ1 How do different neural network operation types impact throughput and energy efficiency on a microcontroller?
• RQ2 To what extent does the distribution of operation types affect the total energy consumption of a neural network, even when total operations and accuracy are similar?
• RQ3 How significant is activation memory compared to model parameters in determining the total memory footprint of a neural network on embedded devices?
• RQ4 Can operation-specific throughput variations invalidate the use of total operations as a proxy for energy efficiency in edge AI models?
• RQ5 How does network topology, particularly feed-forward connections, influence the size of concurrent activation data and overall memory footprint?

Key findings

• Throughput for different neural network operation types varied by up to 5X on an Arm Cortex-M7, with max pooling being the slowest due to branch-heavy implementation.
• Energy per inference varied by as much as 30% across models with similar total operations and accuracy, primarily due to differences in operation type distribution.
• Depthwise separable convolutions had significantly lower throughput than fully-connected or standard convolutions due to lower MACs per output and higher im2col overhead.
• Activation data contributed up to 30% of the total memory footprint in some models, making it a critical factor in memory-constrained deployment.
• Networks with many skip connections, such as DenseNet or NAS-generated architectures, could see activation data increase by up to 10X compared to standard feed-forward networks.
• The number of parameters alone is insufficient to predict memory footprint; concurrent activation data must be considered alongside model weights for accurate estimation.