[Paper Review] Single-Stage 6D Object Pose Estimation

Yinlin Hu, Pascal Fua+2 more|arXiv (Cornell University)|Nov 19, 2019

Robot Manipulation and Learning56 references4 citations

TL;DR

This paper proposes a single-stage 6D object pose estimation framework that directly regresses 6D pose from 3D-to-2D keypoint correspondences using a deep network designed to handle permutation invariance within correspondence groups while respecting keypoint order. By replacing the traditional RANSAC-based PnP step with an end-to-end differentiable network, the method achieves higher accuracy and speed than two-stage baselines, outperforming state-of-the-art models like PoseCNN, SegDriven, and PVNet on LINEMOD and YCB-Video benchmarks.

ABSTRACT

Most recent 6D pose estimation frameworks first rely on a deep network to establish correspondences between 3D object keypoints and 2D image locations and then use a variant of a RANSAC-based Perspective-n-Point (PnP) algorithm. This two-stage process, however, is suboptimal: First, it is not end-to-end trainable. Second, training the deep network relies on a surrogate loss that does not directly reflect the final 6D pose estimation task. In this work, we introduce a deep architecture that directly regresses 6D poses from correspondences. It takes as input a group of candidate correspondences for each 3D keypoint and accounts for the fact that the order of the correspondences within each group is irrelevant, while the order of the groups, that is, of the 3D keypoints, is fixed. Our architecture is generic and can thus be exploited in conjunction with existing correspondence-extraction networks so as to yield single-stage 6D pose estimation frameworks. Our experiments demonstrate that these single-stage frameworks consistently outperform their two-stage counterparts in terms of both accuracy and speed.

Motivation & Objective

• To address the suboptimal training and non-end-to-end nature of two-stage 6D pose estimation pipelines that use RANSAC-based PnP after deep correspondence prediction.
• To eliminate the reliance on surrogate losses (e.g., 2D reprojection error) that do not directly optimize for final pose accuracy.
• To develop a differentiable, end-to-end trainable architecture that directly maps 3D-to-2D correspondences to 6D pose, improving both accuracy and inference speed.
• To demonstrate the generality and effectiveness of the proposed pose regression network when combined with existing correspondence extraction networks.

Proposed method

• The method introduces a deep neural network that takes groups of candidate 3D-to-2D correspondences as input, where each group corresponds to a 3D keypoint.
• The network explicitly models permutation invariance within each group of correspondences (i.e., order of correspondences per 3D keypoint does not affect output), while preserving the fixed order of the groups corresponding to the 3D keypoint sequence.
• It replaces the traditional RANSAC-based PnP algorithm with a learnable, differentiable module that directly regresses the 6D pose from the correspondences.
• The architecture is designed to be generic and can be seamlessly integrated with any state-of-the-art correspondence extraction network, enabling end-to-end training.
• The network is trained using standard optimization with Adam, data augmentation, and a loss function that minimizes 3D and 2D pose errors (ADD and REP) on synthetic and real data.
• The method is evaluated on LINEMOD and YCB-Video datasets using standard metrics: ADD-0.1d and REP-5px.

Experimental results

Research questions

• RQ1 Can a deep network be designed to directly regress 6D pose from 3D-to-2D correspondences in an end-to-end trainable manner, bypassing the need for RANSAC-based PnP?
• RQ2 Does replacing the RANSAC step with a learnable, differentiable module improve pose estimation accuracy and inference speed compared to two-stage frameworks?
• RQ3 Can the proposed pose regression network generalize across different correspondence extraction networks, such as SegDriven and PVNet?
• RQ4 How does the performance of the single-stage framework compare to state-of-the-art two-stage methods in terms of accuracy and runtime under occlusion and clutter?

Key findings

• The single-stage framework, when combined with SegDriven or PVNet for correspondence prediction, achieves consistent improvements in accuracy over the original two-stage versions on the LINEMOD dataset.
• On LINEMOD, the method achieves an ADD-0.1d of 95.3% and REP-5px of 98.7% when combined with PVNet, outperforming the two-stage PVNet baseline.
• On YCB-Video, the method achieves an ADD-0.1d of 89.2% and REP-5px of 94.1%, outperforming PoseCNN, SegDriven, and PVNet in both accuracy and speed.
• The method runs nearly 10 times faster than PoseCNN and almost twice as fast as SegDriven and PVNet due to the elimination of iterative RANSAC.
• The proposed network shows improved repeatability compared to RANSAC, as it is not sensitive to the order of input correspondences, unlike the original PnP step.
• Despite improvements, the learned pose network is less accurate than traditional geometry-based PnP when given very precise correspondences, indicating a gap in performance under ideal conditions.