[Paper Review] HoloLens 2 Research Mode as a Tool for Computer Vision Research

Dorin Ungureanu, Federica Bogo+10 more|arXiv (Cornell University)|Aug 25, 2020

Augmented Reality Applications2 references115 citations

TL;DR

The paper presents HoloLens 2 Research Mode, an API and tools to access raw sensor streams on-device to facilitate computer vision and robotics research, with examples and open-source resources.

ABSTRACT

Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer vision as well as robotics and encourage contributions from the research community.

Motivation & Objective

• Lower the entry barrier for computer vision research in mixed reality by providing raw sensor access on HoloLens 2.
• Introduce a cross-platform API and toolchain to acquire and process sensor data on-device.
• Demonstrate integration with eye and hand tracking to enable richer research workflows.
• Provide open-source samples and documentation to accelerate community contributions.

Proposed method

• Describe the HoloLens 2 hardware and its input streams (VLC cameras, depth, AHAT/Long Throw, IR, IMU).
• Introduce the Research Mode API with a main sensor loop and per-sensor interfaces to retrieve frames and metadata.
• Explain sensor coordinate frames and methods to map between device, world, and camera coordinates using Perception APIs.
• Showcase sample apps from the HoloLens2ForCV repository for visualization, recording, and on-device processing with OpenCV.
• Demonstrate how Research Mode enables CV/ robotics experiments like TSDF integration and SLAM using recorded streams.

Experimental results

Research questions

• RQ1 How can researchers access and synchronize raw HoloLens 2 sensor streams on-device for computer vision experiments?
• RQ2 What are the main novelties of HoloLens 2 Research Mode compared to the first-generation version?
• RQ3 How can Research Mode data be integrated with eye and hand tracking for research workflows?
• RQ4 What sample workflows and tools best demonstrate on-device CV/robotics research on HoloLens 2?
• RQ5 Can researchers reproduce CV/robotics experiments like TSDF integration and SLAM using the provided tools?

Key findings

• HoloLens 2 Research Mode provides APIs to access VLC, depth, AHAT, IR, and IMU streams with frame metadata.
• The API supports per-sensor streams and coordinate transformations to locate sensors relative to the device and world.
• Sample apps demonstrate live visualization, data recording, and synchronization with head, hand, and eye tracking.
• The repository provides examples for TSDF Volume Integration and SLAM using recorded streams and OpenCV-like processing.
• The released tools and documentation foster community contributions to research in computer vision and robotics.