[Paper Review] Merging real and virtual worlds: An analysis of the state of the art and practical evaluation of Microsoft Hololens

Adrien Coppens|arXiv (Cornell University)|Jun 25, 2017

Augmented Reality Applications8 references21 citations

TL;DR

This master's thesis analyzes the state of the art in augmented reality (AR), virtual reality (VR), and mixed reality (MR), with a practical evaluation of Microsoft HoloLens as a leading AR platform. It evaluates motion tracking, computer vision techniques like SLAM and feature detection, and human-computer interaction in AR/VR, concluding with limitations in displays, sensors, interaction, and social acceptance, while proposing future research in adaptive, ubiquitous MR experiences with seamless AR/VR switching.

ABSTRACT

Achieving a symbiotic blending between reality and virtuality is a dream that has been lying in the minds of many people for a long time. Advances in various domains constantly bring us closer to making that dream come true. Augmented reality as well as virtual reality are in fact trending terms and are expected to further progress in the years to come. This master's thesis aims to explore these areas and starts by defining necessary terms such as augmented reality (AR) or virtual reality (VR). Usual taxonomies to classify and compare the corresponding experiences are then discussed. In order to enable those applications, many technical challenges need to be tackled, such as accurate motion tracking with 6 degrees of freedom (positional and rotational), that is necessary for compelling experiences and to prevent user sickness. Additionally, augmented reality experiences typically rely on image processing to position the superimposed content. To do so, "paper" markers or features extracted from the environment are often employed. Both sets of techniques are explored and common solutions and algorithms are presented. After investigating those technical aspects, I carry out an objective comparison of the existing state-of-the-art and state-of-the-practice in those domains, and I discuss present and potential applications in these areas. As a practical validation, I present the results of an application that I have developed using Microsoft HoloLens, one of the more advanced affordable technologies for augmented reality that is available today. Based on the experience and lessons learned during this development, I discuss the limitations of current technologies and present some avenues of future research.

Motivation & Objective

• To analyze the current state of the art in augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies.
• To evaluate Microsoft HoloLens as a practical platform for AR applications, focusing on its technical capabilities and limitations.
• To investigate core enabling technologies such as 6-DoF motion tracking, image processing, and SLAM for spatial-aware AR experiences.
• To identify key challenges in interaction, display quality, sensor performance, and social acceptance for AR/VR adoption.
• To propose future research directions for achieving seamless, ubiquitous blending of reality and virtuality.

Proposed method

• Conducted a systematic review of AR, VR, and MR definitions, taxonomies, and existing applications across domains like healthcare, education, and industry.
• Evaluated technical foundations including mechanical, magnetic, inertial, and optical motion tracking, with emphasis on 6 degrees of freedom (6-DoF) tracking.
• Analyzed vision-based tracking techniques such as template markers (e.g., ARToolkit), natural feature detection (e.g., Harris, FAST), and SLAM for environment understanding.
• Developed a practical AR application using Microsoft HoloLens, named HoloEscape, to test real-world performance and user interaction.
• Compared HoloLens against direct competitors and other MR devices, assessing hardware, spatial awareness, and usability.
• Identified limitations through hands-on development and user testing, focusing on display quality, sensor accuracy, and interaction modalities.

Experimental results

Research questions

• RQ1 How do current AR and VR systems achieve spatial awareness and accurate motion tracking, particularly with 6-DoF?
• RQ2 What are the strengths and weaknesses of vision-based tracking methods such as SLAM, template markers, and natural feature detection in real-world AR applications?
• RQ3 How does the Microsoft HoloLens perform in practical AR application development, and what are its key technical and usability limitations?
• RQ4 What are the main barriers to widespread adoption of AR/VR wearables, including social acceptance and hardware constraints?
• RQ5 What future research directions are most promising for achieving seamless, ubiquitous mixed reality experiences?

Key findings

• The HoloLens enables compelling spatial-aware AR experiences through integrated spatial mapping and 6-DoF tracking, though with limitations in field of view and resolution.
• Vision-based tracking using SLAM and natural features enables markerless AR but remains sensitive to lighting, texture, and motion blur.
• Template markers provide reliable tracking but require predefined markers, limiting flexibility in real-world deployment.
• Current AR/VR interaction methods are limited to gaze, voice, and gesture, with no robust support for complex intent recognition or haptic feedback.
• Social acceptance remains a major barrier due to privacy concerns and the unappealing aesthetic of current wearable devices.
• Future AR/VR systems must evolve toward adaptive, multimodal interaction and seamless switching between AR and VR modes for ubiquitous use.