[Paper Review] Communicating Inferred Goals with Passive Augmented Reality and Active Haptic Feedback

James F. Mullen, Josh Mosier+4 more|arXiv (Cornell University)|Sep 3, 2021

Robot Manipulation and Learning33 references34 citations

TL;DR

This paper proposes a multimodal interface combining passive augmented reality (AR) and active haptic wristbands to improve human-robot teaching during shared autonomy tasks. By passively visualizing inferred goals via AR and actively prompting users with haptic feedback when uncertainty exists, the system reduces interaction time and increases teaching efficiency, outperforming single-modality baselines in a user study.

ABSTRACT

Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arm's motion, the robot gathers information about the human's desired task. But how does the human know what their robot has inferred? Today's approaches often focus on conveying intent: for instance, upon legible motions or gestures to indicate what the robot is planning. However, closing the loop on robot inference requires more than just revealing the robot's current policy: the robot should also display the alternatives it thinks are likely, and prompt the human teacher when additional guidance is necessary. In this paper we propose a multimodal approach for communicating robot inference that combines both passive and active feedback. Specifically, we leverage information-rich augmented reality to passively visualize what the robot has inferred, and attention-grabbing haptic wristbands to actively prompt and direct the human's teaching. We apply our system to shared autonomy tasks where the robot must infer the human's goal in real-time. Within this context, we integrate passive and active modalities into a single algorithmic framework that determines when and which type of feedback to provide. Combining both passive and active feedback experimentally outperforms single modality baselines; during an in-person user study, we demonstrate that our integrated approach increases how efficiently humans teach the robot while simultaneously decreasing the amount of time humans spend interacting with the robot. Videos here: https://youtu.be/swq_u4iIP-g

Motivation & Objective

• Address the lack of feedback in robot learning systems, where humans cannot determine what the robot has inferred about their goals.
• Improve human-robot collaboration by closing the loop on robot inference, not just intent, through multimodal feedback.
• Reduce human monitoring time and increase teaching efficiency by combining passive AR visualization with active haptic prompting.
• Develop a decision framework that dynamically selects between passive and active feedback based on robot confidence and task context.

Proposed method

• Leverages a Microsoft HoloLens for passive AR visualization of multiple likely goal states (e.g., shelf locations) during robot motion.
• Uses wearable haptic wristbands to deliver active, attention-grabbing feedback when the robot is uncertain and requires human input.
• Employs a hybrid feedback algorithm that determines when to display AR visualizations and when to trigger haptic alerts based on robot belief confidence.
• Integrates both modalities into a unified framework for shared autonomy, where the robot infers discrete goals from human teleoperation corrections.
• Designs a decision rule that prioritizes feedback based on uncertainty and potential teaching value, prompting the most informative human input direction.
• Conducts within-subjects user studies comparing AR-only, haptic-only, GUI-only, and AR+Haptic multimodal feedback across real-time teaching tasks.

Experimental results

Research questions

• RQ1 How does combining passive AR and active haptic feedback affect teaching efficiency compared to single-modality feedback?
• RQ2 Can multimodal feedback reduce the time users spend monitoring the robot during teaching tasks?
• RQ3 Does the integration of passive and active feedback improve users’ ability to understand what the robot knows and does not know?
• RQ4 How do users perceive the clarity of intent, prompt timing, and teaching guidance across different feedback modalities?
• RQ5 To what extent does user comfort with AR hardware affect performance and preference in multimodal feedback systems?

Key findings

• The AR+Haptic multimodal system significantly reduced interaction time and increased teaching efficiency compared to all single-modality baselines (p < .01).
• Participants scored statistically significantly higher on the distractor task when using haptic feedback, indicating reduced monitoring time and improved task switching (p < .01).
• Despite AR's passive visualization, it did not significantly reduce interaction time compared to GUI or Haptic conditions (p = .55 and p = .93, respectively), suggesting AR alone is insufficient for reducing user engagement.
• Users reported higher preference for the AR+Haptic condition, citing improved clarity of intent, better awareness of when to teach, and clearer indication of what the robot knew and did not know.
• Subjective ratings showed that AR+Haptic was preferred over alternatives in all five scales (intent clarity, prompt awareness, teaching guidance, knowledge transparency, and overall preference), with all differences being statistically significant (p < .05).
• User discomfort with the HoloLens device may have negatively influenced AR performance, suggesting that GUI-based visual feedback could be a viable alternative to AR in similar systems.