[Paper Review] Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO

Julian Moosmann, Pietro Bonazzi+5 more|arXiv (Cornell University)|Nov 2, 2023

Advanced Memory and Neural Computing4 citations

TL;DR

This paper presents TinyissimoYOLO, a family of sub-million-parameter YOLO-based neural networks optimized for ultra-efficient on-device object detection on AI-integrated smart glasses. Running on the GAP9 RISC-V processor, it achieves 18 FPS with 56ms end-to-end latency and 62.9mW power consumption, enabling up to 9.3 hours of continuous operation on a 154mAh battery, outperforming prior work on similar hardware.

ABSTRACT

Smart glasses are rapidly gaining advanced functions thanks to cutting-edge computing technologies, especially accelerated hardware architectures, and tiny Artificial Intelligence (AI) algorithms. However, integrating AI into smart glasses featuring a small form factor and limited battery capacity remains challenging for a satisfactory user experience. To this end, this paper proposes the design of a smart glasses platform for always-on on-device object detection with an all-day battery lifetime. The proposed platform is based on GAP9, a novel multi-core RISC-V processor from Greenwaves Technologies. Additionally, a family of sub-million parameter TinyissimoYOLO networks are proposed. They are benchmarked on established datasets, capable of differentiating up to 80 classes on MS-COCO. Evaluations on the smart glasses prototype demonstrate TinyissimoYOLO's inference latency of only 17ms and consuming 1.59mJ energy per inference. An end-to-end latency of 56ms is achieved which is equivalent to 18 frames per seconds (FPS) with a total power consumption of 62.9mW. This ensures continuous system runtime of up to 9.3 hours on a 154mAh battery. These results outperform MCUNet (TinyNAS+TinyEngine), which runs a simpler task (image classification) at just 7.3 FPS, while the 18 FPS achieved in this paper even include image-capturing, network inference, and detection post-processing. The algorithm's code is released open with this paper and can be found here: https://github.com/ETH-PBL/TinyissimoYOLO

Motivation & Objective

• To enable always-on, on-device object detection on smart glasses with minimal power consumption and long battery life.
• To address the challenge of deploying complex AI models on resource-constrained, battery-powered wearable devices with small form factors.
• To design a highly efficient neural network architecture tailored for ultra-low-power edge inference on RISC-V-based systems.
• To achieve real-time performance (18 FPS) while integrating image capture, inference, and post-processing within strict energy budgets.
• To demonstrate end-to-end system efficiency by co-optimizing hardware, model architecture, and software stack for smart glasses.

Proposed method

• Proposed TinyissimoYOLO, a family of compact YOLO-based models with fewer than one million parameters, specifically designed for edge deployment.
• Employed neural architecture search and model compression techniques to minimize model size and computational cost while preserving accuracy.
• Integrated the TinyissimoYOLO models into the GAP9 multi-core RISC-V processor, leveraging its ultra-low-power capabilities for on-device inference.
• Optimized the full inference pipeline—including image acquisition, model inference, and post-processing—within a 56ms end-to-end latency budget.
• Used energy-aware model quantization and hardware-aware model compilation to reduce dynamic and static power consumption.
• Validated the system on a physical smart glasses prototype using MS-COCO dataset, achieving 80-class detection with 1.59mJ per inference.

Experimental results

Research questions

• RQ1 Can a sub-million-parameter YOLO-based model achieve real-time object detection (≥18 FPS) on a smart glasses platform with ultra-low power consumption?
• RQ2 How does the end-to-end system latency and energy efficiency compare to existing edge AI solutions on similar hardware?
• RQ3 To what extent can model compression and hardware-aware neural architecture search reduce model size and inference energy without sacrificing detection accuracy?
• RQ4 Can the full pipeline—image capture, inference, and post-processing—be executed within 56ms on a 154mAh battery to enable 9.3 hours of continuous operation?
• RQ5 What is the achievable accuracy and inference speed of TinyissimoYOLO on the MS-COCO dataset when deployed on a real smart glasses prototype?

Key findings

• TinyissimoYOLO achieves 18 FPS with 56ms end-to-end latency on the smart glasses prototype, enabling real-time object detection.
• The system consumes only 62.9mW of total power, allowing for up to 9.3 hours of continuous operation on a 154mAh battery.
• Each inference consumes just 1.59mJ of energy, demonstrating exceptional energy efficiency for on-device object detection.
• The model supports up to 80 object classes on the MS-COCO dataset, maintaining high accuracy despite its compact size.
• The proposed solution outperforms MCUNet (which runs image classification at 7.3 FPS) by achieving higher frame rate while handling a more complex task including post-processing.
• The open-source release of the model code enables reproducibility and further optimization in edge AI research.