[Paper Review] Edge Devices Inference Performance Comparison

Rafał Tobiasz, Grzegorz M. Wilczyński+3 more|arXiv (Cornell University)|Jun 21, 2023

IoT and Edge/Fog ComputingComputer Science3 citations

TL;DR

This paper evaluates the inference performance of popular deep learning models—MobileNet, EfficientNet, ResNet, VGG, and InceptionV3—on four edge platforms: NVIDIA Jetson Nano, Intel Neural Stick, and Google Coral USB and PCIe devices. It focuses on model behavior as feature extractors, measuring inference time across varying input sizes, model scales, and classification head configurations, with key findings showing Google Coral platforms deliver the fastest average inference times, especially for lightweight models like MobileNet and EfficientNet.

ABSTRACT

In this work, we investigate the inference time of the MobileNet family, EfficientNet V1 and V2 family, VGG models, Resnet family, and InceptionV3 on four edge platforms. Specifically NVIDIA Jetson Nano, Intel Neural Stick, Google Coral USB Dongle, and Google Coral PCIe. Our main contribution is a thorough analysis of the aforementioned models in multiple settings, especially as a function of input size, the presence of the classification head, its size, and the scale of the model. Since throughout the industry, those architectures are mainly utilized as feature extractors we put our main focus on analyzing them as such. We show that Google platforms offer the fastest average inference time, especially for newer models like MobileNet or EfficientNet family, while Intel Neural Stick is the most universal accelerator allowing to run most architectures. These results should provide guidance for engineers in the early stages of AI edge systems development. All of them are accessible at https://bulletprove.com/research/edge_inference_results.csv

Motivation & Objective

• To provide a comprehensive performance comparison of popular deep learning models on multiple edge AI platforms.
• To evaluate inference time as a function of input size, model scale, and classification head configuration.
• To focus specifically on models used as feature extractors, which is the primary use case in real-world edge AI systems.
• To guide engineers in selecting optimal model-platform combinations for early-stage development of edge AI systems.
• To share reproducible benchmark data publicly for community use and future research.

Proposed method

• Benchmarked inference performance across four edge devices: NVIDIA Jetson Nano, Intel Neural Stick (NCS2), Google Coral USB, and Google Coral PCIe.
• Evaluated 12 model variants from five families: MobileNet (V1, V2), EfficientNet (V1, V2), ResNet (V1, V2), VGG, and InceptionV3.
• Measured inference time under multiple configurations: input resolution (224×224 and 512×512), presence and size of classification head (1000 or 5 classes), and model scaling (e.g., EfficientNetB0, B1).
• Used TensorFlow Lite with TFLite Interpreter on all platforms, ensuring consistent model deployment and measurement methodology.
• Collected and logged inference times across multiple runs to ensure reliability, with results published at https://bulletprove.com/research/edge_inference_results.csv.
• Addressed hardware limitations such as memory overflow and compilation failures on Coral and Neural Stick platforms during testing.

Experimental results

Research questions

• RQ1 How do different edge AI platforms compare in terms of inference speed for standard computer vision models used as feature extractors?
• RQ2 How does input size (224×224 vs. 512×512) affect inference time across models and platforms?
• RQ3 What is the impact of adding a classification head (1000 or 5 classes) on inference latency?
• RQ4 Which platform offers the best balance of speed, model compatibility, and support for modern architectures like EfficientNetV2?
• RQ5 How do hardware limitations (e.g., memory constraints, unsupported operations) affect model deployment and performance on edge devices?

Key findings

• Google Coral platforms (USB and PCIe) delivered the fastest average inference times, with the Coral PCIe achieving 12.24ms for ResNet50 at 512×512 input size.
• For EfficientNetV2B0 at 512×512, the Coral PCIe achieved an inference time of 16.26ms, significantly outperforming the Jetson Nano (14.43ms) and Neural Stick (5.62ms) in terms of consistency and scalability.
• The Intel Neural Stick was the most universal platform, successfully running all tested models, including large ones like VGG16 and ResNet50, despite higher latency.
• The Jetson Nano exhibited significant latency spikes and was the slowest platform, with inference times exceeding 200ms for VGG16 at 512×512, and was identified as a bottleneck due to model preparation overhead.
• Coral devices showed performance degradation when models exceeded on-chip memory limits, requiring off-chip memory access and increasing latency.
• The Neural Stick failed with large models like VGG16 at 512×512 due to memory constraints, returning an NC_OUT_OF_MEMORY error, while Coral devices handled larger models more robustly.