[Paper Review] POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference

Zhiwen Fan, Panwang Pan+5 more|arXiv (Cornell University)|May 25, 2023

Robot Manipulation and Learning4 citations

TL;DR

POPE introduces a zero-shot 6-DoF object pose estimation framework that achieves accurate relative pose prediction between a single reference image and any target object in arbitrary scenes using a pre-trained 2D foundation model. By combining hierarchical feature representation, 3D geometry principles, and a coarse-to-fine refinement pipeline, POPE reduces median pose error by 52.38% on LINEMOD and 50.47% on OnePose, demonstrating state-of-the-art robustness in open-world, zero-shot settings with minimal supervision.

ABSTRACT

Despite the significant progress in six degrees-of-freedom (6DoF) object pose estimation, existing methods have limited applicability in real-world scenarios involving embodied agents and downstream 3D vision tasks. These limitations mainly come from the necessity of 3D models, closed-category detection, and a large number of densely annotated support views. To mitigate this issue, we propose a general paradigm for object pose estimation, called Promptable Object Pose Estimation (POPE). The proposed approach POPE enables zero-shot 6DoF object pose estimation for any target object in any scene, while only a single reference is adopted as the support view. To achieve this, POPE leverages the power of the pre-trained large-scale 2D foundation model, employs a framework with hierarchical feature representation and 3D geometry principles. Moreover, it estimates the relative camera pose between object prompts and the target object in new views, enabling both two-view and multi-view 6DoF pose estimation tasks. Comprehensive experimental results demonstrate that POPE exhibits unrivaled robust performance in zero-shot settings, by achieving a significant reduction in the averaged Median Pose Error by 52.38% and 50.47% on the LINEMOD and OnePose datasets, respectively. We also conduct more challenging testings in causally captured images (see Figure 1), which further demonstrates the robustness of POPE. Project page can be found with https://paulpanwang.github.io/POPE/.

Motivation & Objective

• To enable 6-DoF object pose estimation for any unseen object in any scene without requiring 3D models or dense annotations.
• To overcome limitations of existing methods that depend on closed-category detection, CAD models, or large sets of posed support views.
• To develop a zero-shot, open-world pose estimation system that generalizes across arbitrary object categories using only a single reference image.
• To improve robustness in cluttered, occluded, or causally captured scenes where traditional 2D-2D matching fails.
• To leverage the zero-shot generalization capability of large-scale 2D foundation models for object retrieval and pose estimation.

Proposed method

• POPE uses a pre-trained 2D foundation model to generate segmentation proposals for objects in new views, enabling open-world detection without category constraints.
• It performs object-level matching between the reference image and segmented proposals using a cross-attention mechanism, enabling zero-shot retrieval of unseen objects.
• The framework estimates relative 6-DoF pose via 2D-2D global matching on coarse correspondences and refines results with 2D-3D local matching using geometric constraints.
• A coarse-to-fine refinement pipeline enables accurate pose estimation across two or more views, even under heavy occlusion or clutter.
• The method integrates hierarchical feature representation and 3D geometry priors to improve matching stability and pose accuracy.
• It supports both two-view and multi-view pose estimation without requiring depth maps or pre-annotated 3D data.

Experimental results

Research questions

• RQ1 Can a single reference image enable accurate 6-DoF pose estimation for any object in an open-world, zero-shot setting?
• RQ2 How can a pre-trained 2D foundation model be leveraged to perform zero-shot object retrieval and relative pose estimation in complex scenes?
• RQ3 What is the performance gain of using a coarse-to-fine 2D-2D and 2D-3D matching pipeline over end-to-end or single-stage methods in zero-shot settings?
• RQ4 How robust is the method in challenging real-world scenarios such as cluttered backgrounds, occlusions, or causally captured images?
• RQ5 Can the framework generalize across diverse object categories without requiring category-specific fine-tuning or 3D models?

Key findings

• POPE reduces the median pose error by 52.38% on the LINEMOD dataset and 50.47% on the OnePose dataset compared to prior methods in zero-shot settings.
• On the OnePose++ dataset, POPE achieves a median error of 6.273, a 42.2% improvement over the next best method (LoFTR) at 9.012.
• In zero-shot two-view pose estimation, POPE achieves 96.2% accuracy at 30° on the OnePose dataset, significantly outperforming LoFTR (96.3%) and Gen6D (89.3%).
• On the LINEMOD dataset, POPE achieves 91.1% accuracy at 15°, surpassing LoFTR (91.8%) and Gen6D (89.3%) in the same setting.
• POPE demonstrates robustness in causally captured images and cluttered scenes, maintaining high accuracy where traditional methods fail.
• The method achieves state-of-the-art performance across all metrics, including median error and accuracy at 15° and 30°, across multiple benchmark datasets with diverse textures and object types.