[Paper Review] DeepMix: Mobility-aware, Lightweight, and Hybrid 3D Object Detection for Headsets

Yongjie Guan, Xueyu Hou+3 more|arXiv (Cornell University)|Jan 15, 2022

Augmented Reality Applications4 citations

TL;DR

DeepMix proposes a lightweight, hybrid 3D object detection framework for mobile headsets that combines edge-offloaded 2D object detection with on-device depth-based 3D bounding box estimation, achieving 30 FPS with end-to-end latency below 100 ms and improving detection accuracy by 9.1–37.3% compared to state-of-the-art models.

ABSTRACT

Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1--37.3% but also reduces end-to-end latency by 2.68--9.15x, compared to the baseline that uses existing 3D object detection models.

Motivation & Objective

• Address the challenge of high computational latency in 3D object detection on resource-constrained mobile headsets for interactive AR/MR applications.
• Overcome the limitations of existing 3D detection models that exceed the 100 ms end-to-end latency threshold required for immersive AR/MR experiences.
• Leverage headset mobility to enhance detection accuracy by refining 3D bounding box estimates through temporal consistency and motion cues.
• Design a lightweight, on-device 3D detection pipeline that minimizes reliance on heavy DNN inference while maintaining high accuracy using depth data.
• Enable real-time, accurate 3D perception on commodity mobile headsets like Microsoft HoloLens 2 without requiring specialized hardware.

Proposed method

• Offload only RGB images to edge servers for fast 2D object detection, reducing local computation load.
• Use the returned 2D bounding box to crop and focus on relevant depth data, drastically reducing the volume of 3D data to process.
• Perform on-device 3D bounding box estimation by analyzing depth frames using geometric constraints and keypoint detection.
• Apply background removal and keypoint projection techniques to refine 3D object dimensions and orientation from depth data.
• Leverage user mobility by fusing multiple observations across frames to improve detection accuracy through temporal consistency.
• Integrate a caching mechanism to optimize depth data access and reduce redundant processing during motion.

Experimental results

Research questions

• RQ1 Can a hybrid 2D/3D detection pipeline reduce end-to-end latency below 100 ms on mobile headsets while maintaining high detection accuracy?
• RQ2 How effective is the exploitation of headset mobility in improving 3D object detection accuracy in dynamic environments?
• RQ3 To what extent can on-device depth processing reduce reliance on computationally heavy 3D DNN inference in AR/MR applications?
• RQ4 How does the combination of edge-based 2D detection and on-device 3D estimation compare to end-to-end 3D detection models in terms of latency and accuracy?
• RQ5 Can a lightweight, mobility-aware 3D detection system achieve real-time performance (30 FPS) on commodity mobile headsets like HoloLens 2?

Key findings

• DeepMix achieves 30 FPS with end-to-end latency below 100 ms, meeting the stringent real-time requirement for interactive AR/MR applications.
• The system improves detection accuracy by 9.1–37.3% compared to baseline 3D object detection models, especially in dynamic and mobile scenarios.
• End-to-end latency is reduced by 2.68–9.15× compared to existing 3D detection models, which typically exhibit 72–283 ms inference times.
• The use of mobility for temporal fusion enhances detection accuracy by leveraging motion to refine 3D bounding box estimates across frames.
• By offloading only 2D detection and processing depth data on-device, DeepMix reduces computational load while maintaining high accuracy.
• User studies with 30+ participants confirm improved user experience, validating the system’s real-time responsiveness and accuracy in immersive AR/MR scenarios.