[Paper Review] Learning Markerless Robot-Depth Camera Calibration and End-Effector Pose Estimation

Bugra C. Sefercik, Barış Akgün|arXiv (Cornell University)|Dec 15, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper presents a learning-based, markerless method for extrinsic calibration between a robot arm and a depth camera using only real-world data, avoiding simulation and fiducial markers. It combines deep learning for end-effector segmentation, rotation prediction, and keypoint detection with ICP refinement and multi-frame fusion to achieve sub-centimeter and sub-degree calibration accuracy, with a single training pose sufficient for high performance.

ABSTRACT

Traditional approaches to extrinsic calibration use fiducial markers and learning-based approaches rely heavily on simulation data. In this work, we present a learning-based markerless extrinsic calibration system that uses a depth camera and does not rely on simulation data. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. We use a transformation trick to get EE pose estimates from rotation predictions and a matching algorithm to get EE pose estimates from keypoint predictions. We further utilize the iterative closest point algorithm, multiple-frames, filtering and outlier detection to increase calibration robustness. Our evaluations with training data from multiple camera poses and test data from previously unseen poses give sub-centimeter and sub-deciradian average calibration and pose estimation errors. We also show that a carefully selected single training pose gives comparable results.

Motivation & Objective

• To eliminate reliance on fiducial markers and simulation data in robot-depth camera extrinsic calibration.
• To enable automatic, real-world data collection and self-supervised labeling for training.
• To achieve high-accuracy end-effector pose estimation and calibration using only depth data and deep learning.
• To develop a robust, reusable calibration system suitable for dynamic or low-cost robotic setups.

Proposed method

• The system uses a MinkUNet18D-based network for semantic segmentation of end-effector, robot, and background from depth point clouds.
• A rotation prediction head estimates the end-effector's orientation using a transformation trick to derive 6-DoF pose from rotation predictions.
• A keypoint detection network identifies 3D landmarks on the end-effector, with pose estimated via orthogonal Procrustes analysis (RPT).
• Multi-frame pose estimates are fused using the Iterative Closest Point (ICP) algorithm to refine calibration and reduce noise.
• Outlier detection and filtering are applied to improve robustness across diverse camera poses and conditions.
• The system collects training data automatically by moving the robot through various joint configurations, with initial calibration used only for data collection.

Experimental results

Research questions

• RQ1 Can a markerless, simulation-free system achieve high-accuracy extrinsic calibration between a robot and depth camera?
• RQ2 Can real-world data alone, without synthetic data, train a model that generalizes across unseen camera poses?
• RQ3 How does multi-frame ICP refinement improve calibration robustness and accuracy?
• RQ4 To what extent can a single training pose yield comparable calibration performance to multiple poses?
• RQ5 Can end-effector segmentation and pose estimation be jointly learned with high accuracy using only depth data?

Key findings

• The system achieves an average calibration error of 0.74 cm and 1.69° across multiple test poses, even when tested on previously unseen camera configurations.
• End-effector pose estimation reaches an average error of 1.0 cm and 2.74°, demonstrating high accuracy in real-time pose estimation.
• A single, carefully selected training pose yields calibration performance comparable to multi-pose training, reducing setup complexity.
• The ICP algorithm significantly improves estimation performance, bringing results close to the physical limit of the depth camera used.
• The system produces high-quality semantic segmentation of the robot and end-effector, enabling downstream perception tasks.
• The method is robust to sensor noise and mechanical errors, provided initial calibration is accurate and forward kinematics errors are accounted for.