[Paper Review] Explainable OOHRI: Communicating Robot Capabilities and Limitations as Augmented Reality Affordances

Lauren W. Wang, Mohamed Kari+1 more|arXiv (Cornell University)|Jan 21, 2026

Social Robot Interaction and HRI0 citations

TL;DR

This work introduces X-OOHRI, an AR interface that communicates robot action possibilities and constraints through visual cues, enabling mixed-initiative interaction and mental model formation. A user study validates its effectiveness.

ABSTRACT

Human interaction is essential for issuing personalized instructions and assisting robots when failure is likely. However, robots remain largely black boxes, offering users little insight into their evolving capabilities and limitations. To address this gap, we present explainable object-oriented HRI (X-OOHRI), an augmented reality (AR) interface that conveys robot action possibilities and constraints through visual signifiers, radial menus, color coding, and explanation tags. Our system encodes object properties and robot limits into object-oriented structures using a vision-language model, allowing explanation generation on the fly and direct manipulation of virtual twins spatially aligned within a simulated environment. We integrate the end-to-end pipeline with a physical robot and showcase diverse use cases ranging from low-level pick-and-place to high-level instructions. Finally, we evaluate X-OOHRI through a user study and find that participants effectively issue object-oriented commands, develop accurate mental models of robot limitations, and engage in mixed-initiative resolution.

Motivation & Objective

• Motivate the need for transparent robot capabilities and limitations in human-robot collaboration.
• Propose an augmented reality interface that conveys object-level robot action possibilities and constraints.
• Enable on-the-fly explanation generation and spatial manipulation of virtual robot twins in a real or simulated environment.

Proposed method

• Encode object properties and robot limits into object-oriented representations using a vision-language model.
• Generate on-demand explanations and present them as AR affordances (visual signifiers, radial menus, color coding, explanation tags).
• Integrate the end-to-end pipeline with a physical robot and a simulated environment for spatial alignment of virtual twins.
• Support a range of tasks from low-level manipulation to high-level instruction through the same framework.

Experimental results

Research questions

• RQ1 How can AR affordances convey robot capabilities and limitations to users in real-time?
• RQ2 Does object-oriented, explainable HRI support accurate mental models of robot behavior?
• RQ3 Can users issue object-oriented commands effectively and engage in mixed-initiative resolution?

Key findings

• Participants effectively issued object-oriented commands using the X-OOHRI interface.
• Users developed accurate mental models of robot limitations.
• The interface supported mixed-initiative problem solving where humans and robots collaborate to resolve tasks.