[Paper Review] Cyber-Physical Testbed for Human-Robot Collaborative Task Planning and Execution

Tuly Hazbar, Shitij Kumar+1 more|arXiv (Cornell University)|May 1, 2019

Human-Automation Interaction and Safety4 citations

TL;DR

This paper presents a cyber-physical testbed integrating a physical tabletop workspace with a real-time digital twin to enable human-robot collaborative task planning and execution. By synchronizing physical actions with virtual state tracking using hierarchical concurrent state machines, the system enables fluent, adaptive collaboration with real-time collision avoidance and dynamic coordination, validated through metrics like concurrent activity and reduced idle time in a 9-piece manipulation task.

ABSTRACT

In this paper, we present a cyber-physical testbed created to enable a human-robot team to perform a shared task in a shared workspace. The testbed is suitable for the implementation of a tabletop manipulation task, a common human-robot collaboration scenario. The testbed integrates elements that exist in the physical and virtual world. In this work, we report the insights we gathered throughout our exploration in understanding and implementing task planning and execution for human-robot team.

Motivation & Objective

• To develop a testbed that supports real-time human-robot collaboration in a shared workspace for tabletop manipulation tasks.
• To investigate how digital twin technology enhances task planning and execution in human-robot teams.
• To evaluate collaboration fluency through objective metrics such as idle time, collisions, and concurrent activity.
• To enable dynamic, adaptive coordination where both human and robot plan and execute actions in real time.
• To provide a framework for designing testbeds that support human-robot fluency and real-time state synchronization.

Proposed method

• The testbed integrates a physical workspace with a real-time digital twin that mirrors the positions of the human, robot, and workpieces using camera-based perception.
• A hierarchical concurrent state machine (HCSM) manages task execution, enabling parallel execution of manipulation and human-robot distance monitoring.
• The system uses motion tracking of the human arm to anticipate actions and adjust robot behavior in real time.
• Collision avoidance is enforced by preempting robot actions when the human-robot distance falls below a predefined threshold.
• Task planning is based on state transitions in the HCSM, with robot actions triggered by human action recognition and goal selection.
• The digital twin synchronizes physical actions with virtual state updates, enabling real-time monitoring and feedback during collaboration.

Experimental results

Research questions

• RQ1 How can a cyber-physical testbed with a digital twin improve real-time human-robot task planning and execution?
• RQ2 What role does real-time human action anticipation play in enabling fluent human-robot collaboration?
• RQ3 How do concurrent execution and dynamic coordination affect task performance and safety?
• RQ4 To what extent can a digital twin enhance the transparency and control of human-robot teaming in shared workspaces?
• RQ5 What metrics best capture the fluency and efficiency of human-robot collaboration in a shared tabletop task?

Key findings

• The digital twin enabled real-time synchronization between physical and virtual states, improving situational awareness and coordination during task execution.
• The system achieved high levels of concurrent activity, with both human and robot actively engaged in 68% of the total task time.
• Robot idle time was reduced to 12% of total task time, indicating effective utilization of robot capabilities.
• Human idle time averaged 15% of total task time, suggesting balanced workload distribution.
• No collisions occurred during the experiments, as the system successfully preempted robot actions when proximity thresholds were breached.
• The task was completed in an average of 4.2 minutes, with both agents performing an average of 11 actions each.