[Paper Review] 3D Robot Pose Estimation from 2D Images

Christoph Heindl, Sebastian Zambal+3 more|arXiv (Cornell University)|Feb 13, 2019

Human Pose and Action Recognition37 references4 citations

TL;DR

This paper proposes an end-to-end deep learning framework for 3D robot pose estimation from single 2D images, jointly predicting joint locations, sparse depth maps, and instance masks using a staged CNN and recurrent segmentation network. Despite being trained exclusively on non-realistic synthetic data, the method generalizes effectively to real-world images, achieving mean pixel errors of 5.81 cm (near) and 9.50 cm (far) for depth estimation.

ABSTRACT

This paper considers the task of locating articulated poses of multiple robots in images. Our approach simultaneously infers the number of robots in a scene, identifies joint locations and estimates sparse depth maps around joint locations. The proposed method applies staged convolutional feature detectors to 2D image inputs and computes robot instance masks using a recurrent network architecture. In addition, regression maps of most likely joint locations in pixel coordinates together with depth information are computed. Compositing 3D robot joint kinematics is accomplished by applying masks to joint readout maps. Our end-to-end formulation is in contrast to previous work in which the composition of robot joints into kinematics is performed in a separate post-processing step. Despite the fact that our models are trained on artificial data, we demonstrate generalizability to real world images.

Motivation & Objective

• To address the lack of large-scale, annotated datasets for robot pose estimation in real-world industrial settings.
• To enable accurate 3D kinematic chain reconstruction from monocular 2D images without requiring 3D depth sensors or manual annotation.
• To develop a method that generalizes from synthetic simulation data to real-world images, avoiding costly data collection and labeling.
• To jointly estimate robot instance segmentation, 2D joint locations, and sparse depth maps in an end-to-end manner.
• To demonstrate that simulation-based training can yield robust performance on real-world robot pose estimation tasks.

Proposed method

• The method uses staged convolutional neural networks to predict belief maps for joint locations and sparse depth maps from a single 2D RGB image.
• A recurrent instance segmentation network processes the belief and depth maps to generate robot instance masks, avoiding complex post-hoc grouping of joints.
• Joint locations are identified via non-maximum suppression on belief maps, and 3D coordinates are reconstructed by re-projection using depth values.
• The model is trained exclusively on artificially rendered 3D simulation data using domain randomization to enhance generalization.
• Instance masks are used to composite joint belief and depth maps per robot, enabling kinematic chain reconstruction.
• The pipeline integrates joint detection, depth regression, and instance segmentation in a single end-to-end framework, eliminating separate post-processing steps.

Experimental results

Research questions

• RQ1 Can a deep learning model trained solely on synthetic 3D-rendered data achieve accurate 3D robot pose estimation on real-world images?
• RQ2 How effective is the joint prediction of 2D joint locations, depth, and instance segmentation in reconstructing complete 3D kinematic chains?
• RQ3 To what extent does the use of a recurrent instance segmentation network improve performance compared to traditional post-hoc grouping of joints?
• RQ4 What is the generalization capability of a simulation-trained model when tested on real-world robot images with varying appearances and poses?
• RQ5 How does the runtime of the pipeline scale with the number of robot instances in a scene?

Key findings

• The model achieves a mean depth estimation error of 5.81 cm in the near range (0–5 m) and 9.50 cm in the far range (5–10 m) on synthetic test data.
• The average precision (AP) at IoU threshold 0.5 is 88.7%, indicating strong instance segmentation performance on synthetic data.
• The method generalizes to real-world images, achieving low pixel errors in joint localization despite no real-world training data.
• Runtime analysis shows that belief prediction is real-time capable (10.95 ms mean), while Laplacian of Gaussian-based joint localization is the slowest component (310.00 ms mean).
• The model successfully reconstructs complete 3D kinematic chains by combining masked joint belief maps and depth maps, even in complex scenes with multiple robots.
• The approach demonstrates that simulation-based training with domain randomization enables effective zero-shot generalization to real-world images, reducing the need for real data collection and annotation.