[Paper Review] Alignment of the Virtual Scene to the Tracking Space of a Mixed Reality Head-Mounted Display

Ehsan Azimi, Qian Long+2 more|arXiv (Cornell University)|Mar 16, 2017

Augmented Reality Applications21 citations

TL;DR

This paper proposes a blackbox calibration method to align the virtual scene coordinate system of an optical see-through HMD with the tracking system's coordinate space, using a 3D projection matrix computed via user alignment. The method achieves up to 4 mm average reprojection error and introduces a faster multipoint calibration requiring only four alignments instead of 20, significantly improving usability and accuracy for mixed reality applications.

ABSTRACT

With the mounting global interest for optical see-through head-mounted displays (OST-HMDs) across medical, industrial and entertainment settings, many systems with different capabilities are rapidly entering the market. Despite such variety, they all require display calibration to create a proper mixed reality environment. With the aid of tracking systems, it is possible to register rendered graphics with tracked objects in the real world. We propose a calibration procedure to properly align the coordinate system of a 3D virtual scene that the user sees with that of the tracker. Our method takes a blackbox approach towards the HMD calibration, where the tracker's data is its input and the 3D coordinates of a virtual object in the observer's eye is the output; the objective is thus to find the 3D projection that aligns the virtual content with its real counterpart. In addition, a faster and more intuitive version of this calibration is introduced in which the user simultaneously aligns multiple points of a single virtual 3D object with its real counterpart; this reduces the number of required repetitions in the alignment from 20 to only 4, which leads to a much easier calibration task for the user. In this paper, both internal (HMD camera) and external tracking systems are studied. We perform experiments with Microsoft HoloLens, taking advantage of its self localization and spatial mapping capabilities to eliminate the requirement for line of sight from the HMD to the object or external tracker. The experimental results indicate an accuracy of up to 4 mm in the average reprojection error based on two separate evaluation methods. We further perform experiments with the internal tracking on the Epson Moverio BT-300 to demonstrate that the method can provide similar results with other HMDs.

Motivation & Objective

• To address the critical challenge of aligning virtual scenes with real-world tracking coordinates in optical see-through HMDs, which is essential for realistic mixed reality experiences.
• To develop a calibration method that is independent of HMD-specific internal parameters, treating the HMD as a blackbox with tracker input and virtual scene output.
• To reduce user fatigue and improve calibration efficiency by introducing a multipoint alignment technique that requires only four alignments instead of 20.
• To enable calibration without line-of-sight constraints by leveraging the HoloLens’ self-localization and spatial mapping capabilities.
• To provide a robust, objective evaluation framework for HMD calibration using an external measurement system and a novel Double-Cube-Match method.

Proposed method

• The method treats the HMD as a blackbox: input is tracker data (e.g., from an external or internal tracking system), and output is the 3D coordinates of a virtual object as perceived by the user.
• A 3D to 3D projection matrix is computed to map the tracker’s coordinate system to the virtual scene’s coordinate system, correcting misalignments between real and virtual content.
• The core technique uses iterative closest point (ICP) or similar optimization to minimize reprojection error between real and virtual points across multiple alignment steps.
• A novel multipoint calibration variant allows simultaneous alignment of multiple 3D points on a single virtual object, reducing the number of required repetitions from 20 to 4.
• The method is validated on Microsoft HoloLens using its internal tracking and spatial mapping, eliminating the need for line-of-sight to external trackers.
• An external measurement system (e.g., motion capture) is used to objectively evaluate calibration accuracy via the Double-Cube-Match evaluation method.

Experimental results

Research questions

• RQ1 How can a consistent and accurate alignment be achieved between the virtual scene and the tracking space of an optical see-through HMD, despite device-specific internal transformations?
• RQ2 Can a blackbox calibration approach eliminate the need for access to internal HMD parameters such as projection matrices?
• RQ3 To what extent does a multipoint alignment technique reduce user effort and improve calibration speed compared to traditional single-point methods?
• RQ4 How does the proposed method perform in terms of reprojection error and user perception accuracy, especially in dynamic or occluded environments?
• RQ5 Can self-localization and spatial mapping features of modern HMDs like the HoloLens be leveraged to eliminate line-of-sight constraints in calibration?

Key findings

• The proposed calibration method achieves an average reprojection error of up to 4 mm, demonstrating high accuracy in aligning virtual content with real-world objects.
• The multipoint calibration variant reduces the number of required alignment steps from 20 to only 4, significantly improving usability and reducing user fatigue.
• The method successfully eliminates the need for line-of-sight between the HMD and external tracker by utilizing the HoloLens’ self-localization and spatial mapping capabilities.
• The Double-Cube-Match evaluation method provides an objective, independent assessment of calibration accuracy, reducing subjectivity in HMD evaluation.
• The method achieves comparable performance on different HMDs, including the HoloLens and Epson Moverio BT-300, confirming its generalizability across platforms.
• The results indicate that the average displacement error is nearly 4 mm, confirming consistency between the two evaluation methods used in the study.