[Paper Review] Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices

Chunjie Luo, Xiwen He+4 more|arXiv (Cornell University)|May 7, 2020

Advanced Neural Network Applications34 references41 citations

TL;DR

AIoTBench benchmarks mobile AI inference across 6 models and 3 frameworks on 5 devices, scoring with VIPS and VOPS to compare AI capability and trade-offs.

ABSTRACT

Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.

Motivation & Objective

• Motivate and define the need for on-device AI inference benchmarking in mobile and embedded devices.
• Propose AIoTBench to evaluate inference across model architectures and frameworks.
• Introduce unified AI scores VIPS and VOPS to compare devices.
• Provide a practical benchmarking workflow on real mobile hardware.

Proposed method

• Define workloads with six networks (three heavy: ResNet50, InceptionV3, DenseNet121; three light: SqueezeNet, MobileNetV2, MnasNet).
• Implement each model in three mobile/embedded frameworks (TensorFlow Lite, Caffe2, PyTorch Mobile).
• Use ImageNet validation set (5 random images per class, total 5000) for inference benchmarking.
• Preprocess inputs with framework-specific normalization, shapes, and color ordering as specified (Table IV).
• Measure accuracy and per-image inference time; compute AI scores VIPS and VOPS via defined equations.

Experimental results

Research questions

• RQ1 How do different AI models trade accuracy, size, and inference speed on mobile devices across frameworks?
• RQ2 How do mobile hardware and frameworks affect on-device AI inference performance?
• RQ3 Do unified VIPS and VOPS scores provide a stable basis for comparing mobile devices?
• RQ4 How does framework selection influence model performance on the same device?
• RQ5 What variability exists in accuracy across devices and implementations for the same model?

Key findings

• AIoTBench covers 6 models across 3 frameworks to benchmark mobile AI inference.
• Two unified metrics VIPS and VOPS summarize end-to-end inference quality and throughput with accuracy as a weight.
• Results show performance varies by device, model, and framework, with no single framework dominating across all conditions.
• Same model and implementation can yield different accuracy on different devices; Oppo R17 shows slight accuracy deviations.
• On different devices, framework performance ordering shifts, e.g., PyTorch Mobile often faster on some devices, while TensorFlow Lite CPU or NNAPI delegation fares differently across models.