[Paper Review] Concepts for End-to-end Augmented Reality based Human-Robot Interaction Systems

David Puljiz, Björn Hein|arXiv (Cornell University)|Oct 10, 2019

Augmented Reality Applications16 references4 citations

TL;DR

This paper presents an end-to-end augmented reality (AR) framework for human-robot interaction using head-mounted displays (HMDs), enabling seamless robot setup, programming, interaction, and imitation learning. By leveraging HMDs like the Microsoft HoloLens for spatial mapping, hand tracking, and real-time referencing, the system achieves robust AR-robot alignment and intuitive collaboration without external sensors, advancing the vision of flexible, collaborative industrial robots.

ABSTRACT

The field of Augmented Reality (AR) based Human Robot Interaction (HRI) has progressed significantly since its inception more than two decades ago. With more advanced devices, particularly head-mounted displays (HMD), freely available programming environments and better connectivity, the possible application space expanded significantly. Here we present concepts and systems currently being developed at our lab to enable a truly end-to-end application of AR in HRI, from setting up the working environment of the robot, through programming and finally interaction with the programmed robot. Relevant papers by other authors will also be overviewed. We demonstrate the use of such technologies with systems not inherently designed to be collaborative, namely industrial manipulators. By trying to make such industrial systems easily-installable, collaborative and interactive, the vision of universal robot co-workers can be pushed one step closer to reality. The main goal of the paper is to provide a short overview of the capabilities of HMD-based HRI to researchers unfamiliar with the concepts. For researchers already using such techniques, the hope is to perhaps introduce some new ideas and to broaden the field of research.

Motivation & Objective

• Address the challenge of creating flexible, reconfigurable, and collaborative robot workcells in industrial environments.
• Overcome limitations of traditional robot cells that require complex setup, fixed infrastructure, and external tracking systems.
• Enable intuitive, AR-based programming and interaction with industrial manipulators using HMDs as the primary interface.
• Integrate AR with robot control stacks (ROS, MoveIt!) to support real-time, spatially-aware human-robot collaboration.
• Advance imitation learning by leveraging HMD sensor data (hand tracking, depth, motion) for natural demonstration and skill acquisition.

Proposed method

• Use HoloLens for self-localization, environmental meshing, and real-time hand and gesture tracking to enable natural interaction.
• Implement a multi-stage referencing pipeline: semi-automatic (seed hologram + ICP/Super4PCS), automatic (descriptor-based matching), and continuous (stereo vision-based tracking).
• Integrate ROS and ROS# via rosbridge to enable bidirectional communication between HMD and robot control systems.
• Apply sensor fusion of HMD hand tracking and wrist-worn IMUs to maintain accurate human pose estimation despite drift.
• Use depth and RGB data from HoloLens to estimate object deformation and contact forces, supporting data collection for imitation learning.
• Develop virtual trajectory programming via AR overlays, allowing users to define waypoints and paths directly in the physical workspace.

Experimental results

Research questions

• RQ1 How can HMD-based AR systems achieve robust, real-time referencing between virtual robot models and physical industrial manipulators without external markers?
• RQ2 To what extent can HMDs replace external tracking systems for human-robot collaboration, and what is the accuracy and reliability of such a solution?
• RQ3 Can AR-based interaction and programming reduce setup time and increase intuitiveness in industrial robot deployment?
• RQ4 How can HMD sensor data be leveraged to support high-fidelity imitation learning for robotic manipulation tasks?
• RQ5 What are the key failure modes and limitations of HMD-based AR in industrial HRI, and how can they be mitigated?

Key findings

• The semi-automatic referencing method using seed holograms and ICP or Super4PCS achieves precise alignment between virtual and real robots, validated through visual and spatial consistency.
• The automatic referencing method using descriptors (OUR-CVFH, SHOT, or custom) enables markerless initialization, though performance varies across robot types and environments.
• Continuous automatic referencing using stereo vision from HoloLens cameras maintains alignment during HMD movement, reducing the need for user re-referencing.
• Sensor fusion of HMD hand tracking and wrist-mounted IMUs enables stable, drift-compensated human pose estimation for real-time collision avoidance.
• HMD-based AR enables intuitive, in-situ robot programming and trajectory definition, significantly improving usability in physical robot workcells.
• HMD sensor data, including depth and motion, supports data collection for imitation learning, with potential for end-to-end learning from human demonstrations in real environments.