[Paper Review] Quantifying the Impact of XR Visual Guidance on User Performance Using a Large-Scale Virtual Assembly Experiment

Leon Pietschmann, Paul-David Zuercher+4 more|arXiv (Cornell University)|Aug 7, 2023

Virtual Reality Applications and Impacts4 citations

TL;DR

This large-scale virtual assembly experiment evaluates XR visual guidance in a web-based immersive environment with 199 participants, demonstrating that XR visual guidance reduces task completion time by 58% and mistakes by 62% compared to no guidance. The study validates and extends Pietschmann et al.'s XR Visual Guidance framework using diverse, real-world-like conditions to quantify performance gains across object identification, action guidance, and gaze guidance modalities.

ABSTRACT

The combination of Visual Guidance and Extended Reality (XR) technology holds the potential to greatly improve the performance of human workforces in numerous areas, particularly industrial environments. Focusing on virtual assembly tasks and making use of different forms of supportive visualisations, this study investigates the potential of XR Visual Guidance. Set in a web-based immersive environment, our results draw from a heterogeneous pool of 199 participants. This research is designed to significantly differ from previous exploratory studies, which yielded conflicting results on user performance and associated human factors. Our results clearly show the advantages of XR Visual Guidance based on an over 50\% reduction in task completion times and mistakes made; this may further be enhanced and refined using specific frameworks and other forms of visualisations/Visual Guidance. Discussing the role of other factors, such as cognitive load, motivation, and usability, this paper also seeks to provide concrete avenues for future research and practical takeaways for practitioners.

Motivation & Objective

• To quantify the impact of different XR visual guidance types—object identification, action guidance, and gaze guidance—on user performance in procedural tasks.
• To address the limitations of prior small-scale, lab-based studies with homogeneous participant pools by conducting a large-scale, diverse, remote experiment.
• To investigate whether performance improvements from XR visual guidance are linked to reduced cognitive load, as commonly hypothesized.
• To refine and field-test the XR Visual Guidance framework proposed by Pietschmann et al. using empirical data from a broad participant base.
• To provide practical design guidelines and future research directions for XR visualisation in industrial and training contexts.

Proposed method

• Conducted a large-scale, web-based virtual assembly experiment using webVR to enable remote participation across diverse devices and environments.
• Designed eight experimental groups with varying combinations of visual guidance: red arrows for object identification, purple lines/green holograms for action guidance, and red edge indicators for gaze guidance.
• Utilized a standardized virtual assembly task with 15 components, requiring sequential placement and connection in a realistic industrial setting.
• Collected objective performance metrics: task completion time and number of mistakes, alongside subjective assessments using NASA-TLX, IMI, and SUS questionnaires.
• Applied statistical analysis to compare performance across visual guidance conditions and assess the influence of human factors such as cognitive load and motivation.
• Implemented technical safeguards including device compatibility checks and data plausibility validation to ensure reliability despite uncontrolled remote settings.

Experimental results

Research questions

• RQ1 How do different forms of XR visual guidance (object identification, action guidance, gaze guidance) affect task completion time and error rates in a virtual assembly task?
• RQ2 To what extent is performance improvement from XR visual guidance associated with reduced cognitive load, as measured by the NASA-TLX?
• RQ3 How do user motivation and perceived usability (measured via IMI and SUS) vary across different visual guidance conditions?
• RQ4 Can the XR Visual Guidance framework by Pietschmann et al. be empirically validated and refined using a large, diverse participant pool in a remote, real-world-like setting?
• RQ5 What are the practical implications and design trade-offs of different visualisation techniques (e.g., arrows vs. glowing highlights) in minimizing visual clutter and occlusion?

Key findings

• XR visual guidance reduced task completion time by 58% compared to the baseline condition with no visual guidance.
• The number of mistakes made during the assembly task was reduced by 62% when XR visual guidance was applied.
• Despite significant performance improvements, no statistically significant difference in cognitive load was observed across visual guidance conditions, challenging the assumption that performance gains stem from reduced mental effort.
• Participants preferred non-occluding visualisation methods such as glowing highlights or color changes over directional arrows, which were perceived as obstructive and disruptive to field of view.
• The study confirmed the effectiveness of the XR Visual Guidance framework in a large-scale, diverse, and remote experimental setting, demonstrating its generalizability beyond small, lab-based studies.
• Subjective feedback highlighted that visual clutter and occlusion from arrows significantly impaired usability, suggesting that alternative highlighting techniques like glow or color shift are more effective in real-world applications.