[Paper Review] ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes

Jianqiu Chen, Zhou, Zikun+5 more|arXiv (Cornell University)|May 29, 2023

Image and Object Detection Techniques4 citations

TL;DR

ZeroPose proposes a fully zero-shot 6D object pose estimation pipeline that leverages CAD models and visual foundation models (SAM and ImageBind) to achieve instance-level segmentation and pose estimation without any training on unseen objects. It achieves state-of-the-art performance with a 25.0% mean average recall (AR) on the BOP benchmark, outperforming prior methods like Megapose by 8.6% in AR while maintaining high efficiency.

ABSTRACT

Many robotics and industry applications have a high demand for the capability to estimate the 6D pose of novel objects from the cluttered scene. However, existing classic pose estimation methods are object-specific, which can only handle the specific objects seen during training. When applied to a novel object, these methods necessitate a cumbersome onboarding process, which involves extensive dataset preparation and model retraining. The extensive duration and resource consumption of onboarding limit their practicality in real-world applications. In this paper, we introduce ZeroPose, a novel zero-shot framework that performs pose estimation following a Discovery-Orientation-Registration (DOR) inference pipeline. This framework generalizes to novel objects without requiring model retraining. Given the CAD model of a novel object, ZeroPose enables in seconds onboarding time to extract visual and geometric embeddings from the CAD model as a prompt. With the prompting of the above embeddings, DOR can discover all related instances and estimate their 6D poses without additional human interaction or presupposing scene conditions. Compared with existing zero-shot methods solved by the render-and-compare paradigm, the DOR pipeline formulates the object pose estimation into a feature-matching problem, which avoids time-consuming online rendering and improves efficiency. Experimental results on the seven datasets show that ZeroPose as a zero-shot method achieves comparable performance with object-specific training methods and outperforms the state-of-the-art zero-shot method with 50x inference speed improvement.

Motivation & Objective

• To address the scalability and high training cost of existing pose estimation methods when applied to unseen objects in industrial robotics and AR applications.
• To enable accurate 6D pose estimation for novel objects without any fine-tuning or retraining, using only a CAD model as input.
• To develop a two-stage pipeline that performs zero-shot instance segmentation and zero-shot pose estimation using visual and geometric priors.
• To improve efficiency and robustness in cluttered scenes by eliminating the need for rendering pose hypotheses or supervised instance detection.
• To establish a fully zero-shot framework that generalizes across diverse, unseen objects in real-world scenarios.

Proposed method

• A two-stage pipeline: first, zero-shot instance segmentation using SAM for candidate mask generation and ImageBind to extract visual embeddings from rendered CAD views for matching.
• Visual descriptor matching via cosine similarity between candidate image features and pre-extracted CAD model features to assign object IDs to segmented instances.
• A lightweight hierarchical geometric structure matching module that directly matches 3D point clouds from the scene to the CAD model’s 3D geometry to estimate 6D pose.
• Pose estimation via 3D-3D point correspondence using RANSAC-based ICP-like matching after filtering noisy points using object radius thresholds.
• The method operates entirely in inference time with no model fine-tuning, relying only on CAD models and visual foundation models.
• The pipeline uses a PyTorch-style pseudocode implementation with uniform sampling of CAD model points and depth projection to generate scene point clouds.

Experimental results

Research questions

• RQ1 Can a CAD model-based zero-shot pose estimation pipeline achieve high accuracy without any training on unseen objects?
• RQ2 Can visual foundation models like SAM and ImageBind be effectively combined to enable zero-shot instance segmentation of novel objects?
• RQ3 Does direct 3D-3D geometric structure matching outperform render-based 2D-2D correspondence methods in terms of accuracy and efficiency?
• RQ4 How does the proposed method compare to SOTA methods like Megapose in zero-shot pose estimation under challenging cluttered scene conditions?
• RQ5 Can the pipeline maintain high performance across diverse, unseen objects in the BOP benchmark datasets?

Key findings

• The proposed zero-shot instance segmentation method achieves comparable performance to supervised MaskRCNN, with a 17.7% mean average recall (AR) on the BOP benchmark.
• The zero-shot pose estimator achieves a 25.0% mean AR across seven BOP core datasets, outperforming Megapose by 8.6 percentage points.
• On LM-O and T-LESS datasets, performance is limited by low signal-to-noise ratio in point clouds, leading to noisy geometric structures and reduced accuracy.
• The method demonstrates superior efficiency by avoiding the generation of pose hypotheses, unlike Megapose, which relies on online rendering and matching.
• The pipeline achieves state-of-the-art results in zero-shot pose estimation while maintaining fast inference, making it suitable for real-time industrial applications.
• The integration of visual foundation models with CAD geometry enables robust, generalizable pose estimation across diverse, unseen objects.