[Paper Review] Mitigating Edge Machine Learning Inference Bottlenecks: An Empirical Study on Accelerating Google Edge Models

Amirali Boroumand, Saugata Ghose+6 more|arXiv (Cornell University)|Mar 1, 2021

Privacy-Preserving Technologies in Data29 references20 citations

TL;DR

This paper proposes Mensa, a heterogeneous ML inference acceleration framework that dynamically schedules layers from diverse Google edge models (CNNs, LSTMs, Transducers, RCNNs) to specialized on-chip and near-data accelerators based on layer characteristics. By clustering layers into three distinct types, Mensa achieves 3.0x higher energy efficiency and 3.1x higher throughput than the Google Edge TPU, and 2.4x/4.3x improvements over Eyeriss v2 across 24 real-world edge models.

ABSTRACT

As the need for edge computing grows, many modern consumer devices now contain edge machine learning (ML) accelerators that can compute a wide range of neural network (NN) models while still fitting within tight resource constraints. We analyze a commercial Edge TPU using 24 Google edge NN models (including CNNs, LSTMs, transducers, and RCNNs), and find that the accelerator suffers from three shortcomings, in terms of computational throughput, energy efficiency, and memory access handling. We comprehensively study the characteristics of each NN layer in all of the Google edge models, and find that these shortcomings arise from the one-size-fits-all approach of the accelerator, as there is a high amount of heterogeneity in key layer characteristics both across different models and across different layers in the same model. We propose a new acceleration framework called Mensa. Mensa incorporates multiple heterogeneous ML edge accelerators (including both on-chip and near-data accelerators), each of which caters to the characteristics of a particular subset of models. At runtime, Mensa schedules each layer to run on the best-suited accelerator, accounting for both efficiency and inter-layer dependencies. As we analyze the Google edge NN models, we discover that all of the layers naturally group into a small number of clusters, which allows us to design an efficient implementation of Mensa for these models with only three specialized accelerators. Averaged across all 24 Google edge models, Mensa improves energy efficiency and throughput by 3.0x and 3.1x over the Edge TPU, and by 2.4x and 4.3x over Eyeriss v2, a state-of-the-art accelerator.

Motivation & Objective

• Identify and analyze the performance bottlenecks in the commercial Google Edge TPU when executing diverse state-of-the-art edge neural network models.
• Quantify the high intra-model and inter-model heterogeneity in layer characteristics (e.g., FLOP/Byte ratio, footprint, data reuse) across CNNs, LSTMs, Transducers, and RCNNs.
• Design a heterogeneous accelerator framework, Mensa, that matches specific accelerators to layer clusters based on their computational and memory access patterns.
• Demonstrate that a minimal three-accelerator design for Mensa can outperform monolithic and reconfigurable accelerators across diverse edge workloads.
• Validate the framework’s effectiveness through comprehensive evaluation on 24 real-world Google edge models, showing significant gains in energy efficiency and throughput.

Proposed method

• Conduct a per-layer analysis of 24 Google edge models to characterize key metrics such as FLOP/Byte ratio, parameter footprint, and data reuse patterns.
• Cluster layers into three distinct groups based on shared characteristics, enabling targeted accelerator specialization.
• Design three heterogeneous accelerators: Pascal (for high-compute, high-reuse layers), Pavlov (for recurrent layers with low reuse), and Jacquard (for low-footprint, high-bandwidth layers).
• Implement a runtime scheduler that assigns each layer to the optimal accelerator, respecting inter-layer dependencies and maximizing utilization.
• Integrate on-chip and near-data processing units (e.g., processing-in-memory) to reduce memory bottlenecks and improve bandwidth efficiency.
• Evaluate Mensa against the Edge TPU and Eyeriss v2 using real workloads, measuring throughput, energy efficiency, and inference latency.

Experimental results

Research questions

• RQ1 What are the primary performance bottlenecks in the Google Edge TPU when executing diverse edge neural network models?
• RQ2 To what extent does intra-model and inter-model heterogeneity in layer characteristics (e.g., FLOP/Byte ratio, footprint, data reuse) impact accelerator efficiency?
• RQ3 Can layer clustering based on shared characteristics enable effective specialization of heterogeneous accelerators for edge ML inference?
• RQ4 How much performance improvement can be achieved by dynamically scheduling layers to purpose-built accelerators compared to monolithic or reconfigurable designs?
• RQ5 What is the optimal number and type of specialized accelerators needed to achieve high performance across diverse edge models?

Key findings

• The Google Edge TPU achieves only 25% average throughput utilization across 24 edge models, with LSTMs and Transducers suffering below 1% utilization due to mismatched dataflow and memory access patterns.
• The Edge TPU operates at only 37% of its theoretical peak energy efficiency on average, primarily due to underutilized on-chip buffers and inefficient memory access handling.
• Layer characteristics vary by up to two orders of magnitude within a single model, especially in edge-optimized models using separable convolutions and recurrent units.
• Mensa improves energy efficiency by 3.0x and throughput by 3.1x over the Edge TPU across all 24 models, with 2.4x and 4.3x gains over Eyeriss v2, respectively.
• LSTMs and Transducers see a 5.4x latency reduction in Mensa due to Pavlov’s optimized dataflow and processing-in-memory support, while CNNs benefit from multi-accelerator load balancing.
• The three-accelerator Mensa design effectively handles all model types with minimal runtime overhead, proving that a small number of specialized units can outperform general-purpose or reconfigurable designs.