[Paper Review] Towards Massive Interaction with Generalist Robotics: A Systematic Review of XR-enabled Remote Human-Robot Interaction Systems

Xian Wang, Luyao Shen+1 more|arXiv (Cornell University)|Mar 18, 2024

Robotics and Automated Systems5 citations

TL;DR

A systematic PRISMA review of XR-enabled remote human-robot interaction (HRI) systems, summarizing XR tech, interaction designs, robot types, tasks, evaluation methods, and future directions.

ABSTRACT

The rising interest of generalist robots seek to create robots with versatility to handle multiple tasks in a variety of environments, and human will interact with such robots through immersive interfaces. In the context of human-robot interaction (HRI), this survey provides an exhaustive review of the applications of extended reality (XR) technologies in the field of remote HRI. We developed a systematic search strategy based on the PRISMA methodology. From the initial 2,561 articles selected, 100 research papers that met our inclusion criteria were included. We categorized and summarized the domain in detail, delving into XR technologies, including augmented reality (AR), virtual reality (VR), and mixed reality (MR), and their applications in facilitating intuitive and effective remote control and interaction with robotic systems. The survey highlights existing articles on the application of XR technologies, user experience enhancement, and various interaction designs for XR in remote HRI, providing insights into current trends and future directions. We also identified potential gaps and opportunities for future research to improve remote HRI systems through XR technology to guide and inform future XR and robotics research.

Motivation & Objective

• Assess how XR technologies (AR/VR/MR) are used to enable remote control of generalist robotics.
• Synthesize interaction modalities, interface designs, and perspectives in XR-based remote HRI.
• Classify robot types and remote tasks studied in XR-enabled HRI literature.
• Evaluate how remote HRI systems are evaluated and where gaps remain.
• Propose a research agenda to advance XR-based remote HRI.

Proposed method

• Follow PRISMA methodology to identify XR-based remote HRI studies.
• Search 2,588 articles across ACM DL, IEEE Xplore, ScienceDirect, plus Google Scholar and Connected Papers.
• Screen and apply inclusion criteria to select 100 articles for data extraction.
• Develop and apply a data extraction rubric covering XR tech, interaction modalities, virtual interfaces, user perspectives, robot types, tasks, evaluation, delays, multiplayer support, and enhancements.
• Perform two-phase data extraction (May 2022 and December 2023) to incorporate latest studies.

Experimental results

Research questions

• RQ1 What XR technologies (VR/AR/MR) are used for remote HRI and in what contexts?
• RQ2 What interaction modalities and virtual interfaces are most effective for remote robot control?
• RQ3 What robot types and remote tasks are investigated in XR-enabled HRI studies, and how are they distributed?
• RQ4 How are XR-based remote HRI systems evaluated, and what delays or performance metrics are reported?
• RQ5 What gaps and future directions emerge for XR-based remote HRI research?

Key findings

• VR HMDs dominate XR usage (67%), followed by AR HMDs (19%), with MR (5%), CAVE (2%), and Mobile AR (1%).
• Most studies use built-in controllers or gestures for interaction; joystick and haptic devices are used in a subset, with 27% using controllers and 20% using gestural methods.
• Robotic arms are the most studied robot type (41%), with navigation, grasping/picking/placement, and industrial tasks as the top activities (26%, 18%, 18%).
• 38% of papers did not include any evaluation; among those that did, 69% used quantitative methods (time/accuracy, NASA-TLX, etc.), and 36% used a single method or a hybrid approach.
• There is a notable concentration of XR-enabled remote HRI research in robotics/manufacturing domains, with a publication peak in 2023.
• The literature presents a taxonomy for XR-based remote HRI interfaces, including user perspectives (robot-coupled, robot-decoupled, bird’s-eye, dynamic) and interface types (direct, digital twin, virtual control room, and hybrid variants).