[Paper Review] CosyPose: Consistent multi-view multi-object 6D pose estimation

Yann Labbé, Justin Carpentier+2 more|arXiv (Cornell University)|Aug 19, 2020

Robotics and Sensor-Based Localization58 references404 citations

TL;DR

CosyPose proposes a multi-view, multi-object 6D object pose estimation framework that jointly estimates camera poses and object 6D poses from RGB images with unknown camera viewpoints. It uses a render-and-compare single-view pose estimator, robust RANSAC-based object-level matching across views, and global object-level bundle adjustment to achieve consistent, accurate scene reconstruction without depth sensors, outperforming state-of-the-art methods on YCB-Video and T-LESS by large margins.

ABSTRACT

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed CosyPose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage https://www.di.ens.fr/willow/research/cosypose/.

Motivation & Objective

• To address the challenge of estimating consistent 6D poses of multiple known objects in a scene from multiple RGB images with unknown camera poses.
• To handle object symmetries, missing or incorrect detections, and unknown object counts without requiring depth sensors.
• To improve single-view 6D pose estimation accuracy through a novel render-and-compare approach.
• To jointly refine camera and object poses across views using a global optimization strategy.
• To achieve robustness against false positives and distractors in complex scenes.

Proposed method

• A single-view 6D pose estimation method based on a render-and-compare approach inspired by DeepIM, generating 6D pose hypotheses from RGB images.
• A robust, RANSAC-based object-level matching procedure that matches 6D pose hypotheses across views to estimate relative camera poses and identify consistent object instances.
• A global refinement step using object-level bundle adjustment that minimizes reprojection error across all views to improve pose accuracy.
• A 3D non-maximum suppression (NMS) strategy to resolve duplicate object hypotheses and retain the most consistent candidate.
• A method to automatically recover the number of objects in the scene by analyzing consistent hypothesis matches.
• A robust filtering mechanism that marks inconsistent pose hypotheses (e.g., from distractors) as outliers, improving generalization to unseen objects.

Experimental results

Research questions

• RQ1 Can a multi-view, multi-object 6D pose estimation system achieve consistent scene reconstruction without known camera poses or depth data?
• RQ2 How can inconsistent or incorrect 6D pose hypotheses from single-view networks be filtered out using cross-view consistency?
• RQ3 To what extent does global bundle adjustment improve pose accuracy when combining multiple views?
• RQ4 How does the method handle symmetric, textureless, or partially occluded objects in complex scenes?
• RQ5 Can the system automatically detect and reject false positive detections from distractor objects not in the object database?

Key findings

• CosyPose achieves a 34.2% absolute improvement over the previous state-of-the-art on the T-LESS dataset, demonstrating significant gains in 6D pose estimation accuracy.
• The multi-view framework outperforms [20] on YCB-Video despite not requiring known camera poses or limiting to one object per class.
• The method successfully filters out false positive detections from distractor objects by identifying inconsistent pose hypotheses across views, as shown in visualizations and quantitative results.
• The use of 3D NMS effectively resolves duplicate object hypotheses, retaining only the most consistent 6D pose estimate per object instance.
• The system correctly reconstructs scenes even when some views contain only two visible objects, provided at least three consistent object candidates exist across views.
• Limitations such as consistent false positives (e.g., from similar viewpoints) and missing detections in sparse views are identified and analyzed, with potential mitigation strategies proposed.