[Paper Review] Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking

Kai Chen, Rui Cao+5 more|arXiv (Cornell University)|Apr 14, 2022

Robot Manipulation and Learning4 citations

TL;DR

This paper proposes an iterative self-training framework for sim-to-real 6D object pose estimation in robotic bin-picking, leveraging a photo-realistic simulator to generate synthetic data and iteratively refining pose estimation on real data using pseudo-labels from a teacher model. The method achieves a 22.62% improvement in ADD(-S) on a new dataset and boosts robotic bin-picking success by 19.54%, demonstrating effective domain adaptation without manual annotation.

ABSTRACT

In this paper, we propose an iterative self-training framework for sim-to-real 6D object pose estimation to facilitate cost-effective robotic grasping. Given a bin-picking scenario, we establish a photo-realistic simulator to synthesize abundant virtual data, and use this to train an initial pose estimation network. This network then takes the role of a teacher model, which generates pose predictions for unlabeled real data. With these predictions, we further design a comprehensive adaptive selection scheme to distinguish reliable results, and leverage them as pseudo labels to update a student model for pose estimation on real data. To continuously improve the quality of pseudo labels, we iterate the above steps by taking the trained student model as a new teacher and re-label real data using the refined teacher model. We evaluate our method on a public benchmark and our newly-released dataset, achieving an ADD(-S) improvement of 11.49% and 22.62% respectively. Our method is also able to improve robotic bin-picking success by 19.54%, demonstrating the potential of iterative sim-to-real solutions for robotic applications.

Motivation & Objective

• To address the high cost and difficulty of annotating real-world 6D object pose data for robotic bin-picking applications.
• To reduce the sim-to-real domain gap in 6D object pose estimation without requiring custom simulation setups or network architectures.
• To enable scalable, network-agnostic adaptation of pose estimation models to new real-world bin-picking scenarios using only unlabeled real data.
• To improve real-world performance of pose estimation models trained solely on synthetic data through iterative self-training with pseudo-labeling.

Proposed method

• A photo-realistic simulator is used to generate abundant synthetic RGB-D data with accurate 6D pose annotations for pre-training an initial pose estimation network.
• The pre-trained model acts as a teacher to predict 6D poses on unlabeled real-world data, generating pseudo-labels for self-training.
• A comprehensive adaptive selection scheme combines 2D appearance and 3D geometric cues to identify reliable pseudo-labels and filter out noisy predictions.
• The student model is fine-tuned on the selected pseudo-labeled real data, and the process is iteratively repeated with the updated student as the new teacher.
• The iterative refinement process progressively improves pose estimation accuracy on real data, with performance saturating after ~5 iterations.
• The method is evaluated on a public benchmark and a newly released dataset, and deployed on a real robot arm for bin-picking tasks.

Experimental results

Research questions

• RQ1 Can iterative self-training with pseudo-labeling effectively close the sim-to-real gap in 6D object pose estimation without manual annotation?
• RQ2 How does the performance of the pose estimation model improve with increasing iterations of self-training on real data?
• RQ3 To what extent can this method generalize across different backbone networks and real-world bin-picking scenarios?
• RQ4 Can the proposed method significantly improve robotic bin-picking success rates by enhancing pose estimation accuracy?

Key findings

• The proposed method achieves an 11.49% improvement in ADD(-S) on the public ROBI benchmark dataset.
• On the newly released dataset, the method improves ADD(-S) by 22.62%, demonstrating strong domain adaptation capability.
• Robotic bin-picking success rate increases from 67.96% to 87.50% after applying the self-training framework.
• The first iteration yields the largest performance gain, with diminishing returns after approximately five iterations.
• The method generalizes across different backbone networks, such as DenseFusion, showing consistent improvement on real data.
• Qualitative results show that iterative refinement leads to pose predictions that visually converge toward ground truth over time.