[Paper Review] PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation

Thomas Jantos, Mohamed Amin Hamdad+3 more|arXiv (Cornell University)|Nov 25, 2022

Robot Manipulation and Learning28 citations

TL;DR

PoET is a transformer-based RGB-only method that estimates 6D poses for multiple objects in a single image without depth or 3D models, achieving state-of-the-art results on YCB-V.

ABSTRACT

Accurate 6D object pose estimation is an important task for a variety of robotic applications such as grasping or localization. It is a challenging task due to object symmetries, clutter and occlusion, but it becomes more challenging when additional information, such as depth and 3D models, is not provided. We present a transformer-based approach that takes an RGB image as input and predicts a 6D pose for each object in the image. Besides the image, our network does not require any additional information such as depth maps or 3D object models. First, the image is passed through an object detector to generate feature maps and to detect objects. Then, the feature maps are fed into a transformer with the detected bounding boxes as additional information. Afterwards, the output object queries are processed by a separate translation and rotation head. We achieve state-of-the-art results for RGB-only approaches on the challenging YCB-V dataset. We illustrate the suitability of the resulting model as pose sensor for a 6-DoF state estimation task. Code is available at https://github.com/aau-cns/poet.

Motivation & Objective

• Motivate accurate 6D pose estimation for robotics tasks (grasping, localization) using only RGB images.
• Develop a backbone-agnostic framework that can add 6D pose estimation on top of any 2D object detector.
• Incorporate global image context through a transformer to predict per-object translation and rotation without extra inputs.
• Evaluate the approach on the YCB-V dataset and demonstrate its use as a pose sensor for 6-DoF localization.

Proposed method

• Process an RGB image with a backbone object detector to produce multi-scale feature maps and object bounding boxes.
• Use a deformable transformer encoder–decoder where bounding box information becomes object queries for the decoder.
• Feed the transformer outputs into separate translation and rotation heads to predict 3D translation and 6D rotation (using the 6D rotation representation and geodesic loss).
• Train with a weighted multi-task loss combining translation L2 loss and geodesic rotation loss.
• Support class-specific or class-agnostic setups by adjusting output dimensions to 3n_cls and 6n_cls respectively.
• Train end-to-end on RGB data without depth, 3D models, or symmetry priors, and optionally with ground-truth ROIs for evaluation.

Experimental results

Research questions

• RQ1 Can global image context learned by a transformer improve RGB-only 6D pose estimation for multiple objects in a single view?
• RQ2 Is it possible to predict per-object 3D translation and 6D rotation from RGB alone, using only bounding box information as guidance, without 3D models or depth maps?
• RQ3 How does PoET perform relative to RGB-based state-of-the-art methods on YCB-V with respect to translation, rotation, and ADD-S metrics?
• RQ4 Can PoET serve effectively as a pose sensor for 6-DoF camera localization in cluttered scenes?

Key findings

• PoET achieves state-of-the-art results among RGB-only methods on YCB-V (ADD-S metric).
• The model handles multiple objects in a single RGB image without iterative refinement or 3D models.
• Using bounding box information as part of the transformer input improves learning and performance.
• The translation and rotation heads produce competitive translation accuracy and rotation accuracy, with the rotation being competitive even without symmetry priors.
• Ablation studies show the importance of bounding box-guided queries and the full-image context for best performance.
• PoET can be used for camera localization by fusing relative pose estimates from detected objects.