[Paper Review] Deep Model-Based 6D Pose Refinement in RGB

Fabian Manhardt, Wadim Kehl+2 more|arXiv (Cornell University)|Oct 7, 2018

Advanced Vision and Imaging32 references4 citations

TL;DR

This paper proposes a deep learning-based method for 6D pose refinement in RGB images using a differentiable visual loss that aligns object contours without requiring correspondences, segmentation, or explicit appearance modeling. The approach achieves near-ICP accuracy in pose estimation using only synthetic RGB data and runs in real time, demonstrating robustness to occlusion, symmetry, and poor initialization.

ABSTRACT

We present a novel approach for model-based 6D pose refinement in color data. Building on the established idea of contour-based pose tracking, we teach a deep neural network to predict a translational and rotational update. At the core, we propose a new visual loss that drives the pose update by aligning object contours, thus avoiding the definition of any explicit appearance model. In contrast to previous work our method is correspondence-free, segmentation-free, can handle occlusion and is agnostic to geometrical symmetry as well as visual ambiguities. Additionally, we observe a strong robustness towards rough initialization. The approach can run in real-time and produces pose accuracies that come close to 3D ICP without the need for depth data. Furthermore, our networks are trained from purely synthetic data and will be published together with the refinement code to ensure reproducibility.

Motivation & Objective

• Address the challenge of 6D pose refinement in RGB-only scenarios where depth data is unavailable.
• Overcome limitations of existing methods that rely on hand-crafted segmentation, correspondences, or appearance models.
• Enable robust pose refinement under occlusion, symmetry, and visual ambiguities without manual tuning.
• Achieve real-time performance while maintaining high accuracy comparable to depth-based ICP methods.
• Train the network entirely on synthetic data to ensure generalization and reproducibility.

Proposed method

• Propose a differentiable visual loss that drives pose refinement by maximizing overlap between rendered object contours and scene contours.
• Train a CNN to predict 6D pose updates (translation and rotation) based on RGB input and synthetic renderings.
• Use synthetic RGB data with varying illumination and occlusion to implicitly learn object appearance and shape during training.
• Avoid explicit modeling of appearance or geometry by relying solely on contour alignment as the optimization signal.
• Design a parameter-free optimization framework that does not require RANSAC, segmentation, or correspondence matching.
• Ensure generalization by training on diverse CAD models and evaluating on unseen instances from the same category.

Experimental results

Research questions

• RQ1 Can a deep neural network refine 6D object poses in RGB using only contour alignment, without explicit appearance or correspondence modeling?
• RQ2 To what extent can a network trained on synthetic data generalize to real-world RGB images with occlusion and visual ambiguity?
• RQ3 How does contour-based learning compare to traditional ICP or appearance-based methods in terms of accuracy and robustness?
• RQ4 Can the method handle symmetric objects and strong initialization errors without manual intervention?
• RQ5 Does the absence of depth data significantly degrade performance compared to RGB-D methods?

Key findings

• The method achieves pose accuracy close to 3D ICP without requiring depth data, demonstrating strong performance in RGB-only scenarios.
• On the Occlusion dataset, the ADD score improved from 6.2% to 28.5%, indicating significant robustness to partial occlusion.
• Translational error decreased by 4mm along X and Y axes and 28mm along Z-axis compared to baseline methods under occlusion.
• The network trained with the visual loss outperformed MSE-based training by 14% in VSS metric on symmetric objects, highlighting the importance of proper symmetry handling.
• The approach generalizes to category-level tracking, successfully refining poses for previously unseen mug and bowl models after training on a diverse set of similar CAD models.
• Failure cases include misidentification of similarly shaped or colored objects and incomplete recovery under heavy occlusion, indicating current limitations in complex visual confusion.