[Paper Review] Toward safe separation distance monitoring from RGB-D sensors in human-robot interaction

Petr Švarný, Zdeněk Straka+1 more|arXiv (Cornell University)|Oct 11, 2018

Robotic Path Planning Algorithms8 references4 citations

TL;DR

This paper presents a real-time framework for safe separation distance monitoring between humans and robots using RGB-D sensors and human pose estimation. By computing pairwise distances between body keypoints on the human and robot, the system enables dynamic, tunable safety monitoring beyond fixed zones, demonstrated on a Nao robot with RealSense and OpenPose.

ABSTRACT

The interaction of humans and robots in less constrained environments gains a lot of attention lately and the safety of such interaction is of utmost importance. Two ways of risk assessment are prescribed by recent safety standards: (i) power and force limiting and (ii) speed and separation monitoring. Unlike typical solutions in the industry that are restricted to mere safety zone monitoring, we present a framework that realizes separation distance monitoring between a robot and a human operator in a detailed, yet versatile, transparent, and tunable fashion. The separation distance is assessed pair-wise for all keypoints on the robot and the human body and as such can be selectively modified to account for specific conditions. The operation of this framework is illustrated on a Nao humanoid robot interacting with a human partner perceived by a RealSense RGB-D sensor and employing the OpenPose human skeleton estimation algorithm.

Motivation & Objective

• Address the growing need for safe human-robot interaction (HRI) in unstructured environments.
• Overcome limitations of traditional safety zone monitoring by enabling fine-grained, dynamic separation distance tracking.
• Develop a transparent and configurable method for assessing inter-agent distances in real time.
• Integrate human pose estimation with robot kinematics to enable precise, pairwise distance computation.
• Demonstrate the framework’s feasibility and responsiveness in a real-world HRI scenario with a Nao robot.

Proposed method

• Utilizes a RealSense RGB-D sensor to capture depth and color data for human and robot perception.
• Employs the OpenPose algorithm to estimate 2D/3D human body keypoints in real time.
• Maps robot joint states to 3D keypoints using forward kinematics for consistent comparison.
• Computes pairwise Euclidean distances between all human and robot body keypoints to assess proximity.
• Applies configurable thresholds per keypoint pair to enable tunable safety policies.
• Integrates the framework into a real-time control loop for dynamic response to proximity changes.

Experimental results

Research questions

• RQ1 How can separation distance between humans and robots be monitored with fine-grained, real-time precision beyond fixed safety zones?
• RQ2 Can RGB-D sensors and human pose estimation enable reliable, low-latency inter-agent distance tracking in HRI?
• RQ3 To what extent can safety monitoring be made transparent and configurable through keypoint-level distance assessment?
• RQ4 How does the system perform in a real-world HRI scenario with a humanoid robot and dynamic human motion?
• RQ5 Can the framework support selective safety policies based on anatomical regions or task-specific constraints?

Key findings

• The framework enables real-time, pairwise distance monitoring between human and robot body keypoints using only RGB-D data and pose estimation.
• The system supports configurable safety thresholds per keypoint pair, allowing tailored responses to different interaction scenarios.
• The approach achieves sufficient temporal resolution and accuracy for dynamic HRI, as demonstrated in a live interaction with a Nao robot.
• The integration of OpenPose with robot kinematics allows consistent 3D keypoint alignment for accurate distance computation.
• The method provides a transparent and tunable alternative to fixed safety zones, enhancing adaptability in unstructured environments.
• The implementation demonstrates feasibility and responsiveness in a real-world HRI setting, supporting future safety-critical applications.