[Paper Review] ARGUS: Visualization of AI-Assisted Task Guidance in AR

Sonia Castelo, João Rulff+16 more|arXiv (Cornell University)|Aug 11, 2023

Augmented Reality ApplicationsComputer Science3 citations

TL;DR

ARGUS is a visual analytics system that enables real-time and retrospective analysis of multimodal sensor data and AI model outputs in AR-assisted task guidance. It supports developers in debugging, improving, and fine-tuning AI assistants by providing interactive 3D and temporal visualizations of object, action, and step detection, gaze tracking, and model confidence across complex AR sessions.

ABSTRACT

The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine tune the components of the AR assistant.

Motivation & Objective

• To address the challenge of developing reliable, real-time AI-assisted AR assistants that perceive the environment, reason about tasks, and adapt to user behavior.
• To support ML and AR developers in troubleshooting and improving AI models used in AR task guidance by visualizing complex, multimodal sensor and model output data.
• To enable retrospective analysis of AR sessions to understand model behavior, performer actions, and attention patterns over time.
• To reduce the cognitive load on ML engineers by providing scalable, interactive visualizations that contextualize model predictions in spatial and temporal dimensions.
• To support the co-design of AR assistants by integrating insights from visualization, machine learning, and AR research communities.

Proposed method

• ARGUS integrates real-time and offline visualization of multimodal data streams from AR headsets, including video, depth, gaze, audio, and time-of-flight sensors.
• The system provides a 3D spatial view that overlays performer gaze projections and object detections onto a reconstructed world point cloud.
• Temporal and spatial visualization widgets allow users to explore model outputs, such as object and action detection, step detection, and confidence scores, across the entire session.
• The system supports selective visualization of model outputs and sensor data at varying granularities, enabling users to inspect predictions in context with underlying video frames.
• It includes tools for identifying and excluding noisy data, such as hand artifacts in point clouds, through manual bounding box definitions and future integration of automated noise removal.
• ARGUS enables annotation of data and model outputs, with plans to extend support for multi-performer session comparison and privacy-preserving data handling.

Experimental results

Research questions

• RQ1 How can visual analytics tools support developers in understanding and debugging AI models within real-time AR task guidance systems?
• RQ2 What visualization techniques best enable the exploration of spatiotemporal relationships between performer actions, sensor data, and model outputs in AR?
• RQ3 How can interactive 3D and temporal visualizations improve insight into model confidence, prediction errors, and attention patterns during AR task execution?
• RQ4 What role does contextual, multi-modal data integration play in enabling effective model refinement and system improvement?
• RQ5 How can visual analytics systems like ARGUS support the co-design of AR assistants across visualization, ML, and AR research domains?

Key findings

• ARGUS enables developers to identify and resolve model prediction errors by visualizing object and action detection outputs in spatial and temporal context, such as detecting that 'Plate' was only recognized late in a task.
• The system revealed that gaze heatmaps effectively highlight performer attention patterns, such as the final focus on a recipe, aiding in understanding task engagement and model alignment.
• Interactive visualization of model confidence scores helped ML engineers diagnose underperformance in specific conditions, informing data augmentation and model retraining strategies.
• The integration of 3D spatial visualization with temporal tracking allowed for better understanding of how model predictions change based on viewpoint and environmental context.
• The system reduced the burden on ML engineers by replacing terminal logging and video overlay techniques with a scalable, interactive interface for model analysis.
• User feedback from the co-design process demonstrated that ARGUS improved model interpretability and facilitated faster iteration cycles in AR assistant development.