[Paper Review] MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare

Yann Labbé, Lucas Manuelli+8 more|arXiv (Cornell University)|Dec 13, 2022

Robot Manipulation and LearningEngineering68 citations

TL;DR

MegaPose estimates the 6D pose of novel objects at inference time by using a render-and-compare refinement pipeline and a novel coarse pose estimator, trained on a large synthetic dataset, achieving state-of-the-art results on ModelNet and competitive results on BOP without object-specific training.

ABSTRACT

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

Motivation & Objective

• Enable 6D pose estimation for novel objects using only a CAD model and an image region of interest.
• Develop a coarse pose estimator that functions without object-specific symmetry knowledge during training.
• Design a refinement network that can generalize to unseen objects by rendering multiple synthetic views.
• Leverage a large-scale synthetic dataset to improve generalization to novel objects.
• Evaluate on ModelNet and BOP datasets to demonstrate state-of-the-art or competitive performance.

Proposed method

• Coarse pose estimation via a classification model that selects the rendered view whose pose is most compatible with the observed image, enabling handling of symmetries.
• Pose refinement network that, given an initial pose, renders multiple views with an anchor point and predicts pose updates in a disentan gled rotation/translation formulation.
• Anchor point handling by projecting the anchor to the image center and using multiple viewpoints to infer its location, enabling generalization to novel objects.
• Input can be RGB or RGB-D, with a ResNet-34 backbone followed by a single fully-connected layer for the coarse model and a 9-dimension output for the refiner.
• Training on a large synthetic dataset (2 million images) generated from ShapeNet and Google Scanned Objects, with data augmentation and depth/normal channels to improve realism and occlusion robustness.
• Depth/normal channels are included and depth inputs are normalized to assist scale-generalization across objects.

Experimental results

Research questions

• RQ1 Can a 6D pose estimator be trained to generalize to novel objects not seen during training by using synthetic data and render-and-compare refinement?
• RQ2 Can a coarse pose estimator trained without explicit symmetry information effectively initialize refinements for novel objects?
• RQ3 How does large-scale synthetic data influence generalization to novel object instances across diverse benchmarks?
• RQ4 What is the performance of MegaPose on standard 6D pose benchmarks (ModelNet, YCB-Video, and the seven BOP core datasets) when objects are novel at test time?

Key findings

• The coarse+refiner pipeline significantly improves initial pose estimates and achieves competitive or superior results on several benchmarks while not requiring training on target objects.
• The RGB-D variant of MegaPose yields notable AR improvements on YCB-V and LM-O compared to prior methods like Zephyr, with an average improvement reported on the benchmark pair.
• The refiner delivers substantial gains (e.g., +41.0 AR on RGB-D) and runs at about 50 ms per iteration on a RTX 2080, enabling near real-time refinement.
• Training on a large, diverse synthetic dataset greatly improves generalization to novel objects, with ablations showing performance gains as the number and variety of training objects increase.
• On ModelNet, the MegaPose refiner outperforms state-of-the-art methods such as MP-AAE and LatentFusion across multiple metrics.
• Across the seven BOP core datasets, MegaPose demonstrates strong performance, competitive with methods that require access to target objects during training.