[Paper Review] I Like to Move It: 6D Pose Estimation as an Action Decision Process

Benjamin Busam, Hyun Jun Jung+1 more|arXiv (Cornell University)|Sep 26, 2020

Robot Manipulation and LearningEngineering76 references18 citations

TL;DR

This paper proposes a novel 6D object pose estimation method that frames the task as an iterative action decision process using a lightweight CNN to predict incremental pose corrections from a single RGB image. By refining the pose through sequential, learned actions and dynamically adjusting inference cost based on motion, the approach achieves state-of-the-art accuracy on YCB and Laval datasets while generalizing to unseen objects without fine-tuning.

ABSTRACT

Object pose estimation is an integral part of robot vision and AR. Previous 6D pose retrieval pipelines treat the problem either as a regression task or discretize the pose space to classify. We change this paradigm and reformulate the problem as an action decision process where an initial pose is updated in incremental discrete steps that sequentially move a virtual 3D rendering towards the correct solution. A neural network estimates likely moves from a single RGB image iteratively and determines so an acceptable final pose. In comparison to other approaches that train object-specific pose models, we learn a decision process. This allows for a lightweight architecture while it naturally generalizes to unseen objects. A coherent stop action for process termination enables dynamic reduction of the computation cost if there are insignificant changes in a video sequence. Instead of a static inference time, we thereby automatically increase the runtime depending on the object motion. Robustness and accuracy of our action decision network are evaluated on Laval and YCB video scenes where we significantly improve the state-of-the-art.

Motivation & Objective

• To address the limitations of existing 6D pose estimation methods that require object-specific training and suffer from poor generalization.
• To improve robustness and accuracy in the presence of noise, occlusion, and clutter using a single RGB image input.
• To reduce computational cost by dynamically adjusting inference complexity based on object motion and pose change.
• To enable zero-shot generalization to unseen objects by learning a decision process rather than a per-object model.
• To leverage video sequences for iterative refinement, improving tracking performance and reducing runtime through pose initialization from previous frames.

Proposed method

• The method formulates 6D pose estimation as a sequential decision process where a neural network predicts discrete pose actions to iteratively refine a 3D object rendering.
• A lightweight CNN processes cropped image and rendering pairs to predict a 6D pose update (translation and rotation) in each step.
• The predicted action is applied to the current pose, and a new rendering is generated and re-cropped for the next iteration.
• A stop action is predicted when pose changes become insignificant, enabling dynamic reduction of inference cost in video sequences.
• The network is trained on synthetic data with high-quality rendering on real backgrounds, using a data augmentation scheme to simulate clutter and occlusion.
• For video inference, the pose from frame t is used as the initial pose for frame t+1, reducing the number of required iterations.

Experimental results

Research questions

• RQ1 Can 6D pose estimation be effectively reformulated as a sequential action decision process rather than regression or classification?
• RQ2 Does a lightweight, shared policy network generalize better to unseen objects than object-specific models?
• RQ3 Can dynamic complexity reduction based on pose change improve inference efficiency in video sequences?
• RQ4 How does the method perform under varying levels of occlusion, clutter, and noise compared to state-of-the-art methods?
• RQ5 Can a single RGB image input with iterative refinement outperform existing methods on benchmark datasets like YCB and Laval?

Key findings

• The proposed method achieves state-of-the-art performance on the YCB dataset, with an average AUC of 83.47% across all objects under the ADD threshold curve for 0.1m.
• On the Laval dataset, the method achieves a mean translation error of 3.82mm and 6.48° for the cookie jar under 0% occlusion, improving significantly over prior work.
• The model without rendered depth input (Ours w/o D) shows higher errors, indicating that depth supervision improves robustness, especially under occlusion.
• The method generalizes to unseen objects without fine-tuning, demonstrating zero-shot capability due to the learned decision policy.
• In video sequences, the dynamic complexity reduction scheme reduces inference cost by leveraging initial poses from previous frames, with fewer iterations needed when motion is low.
• The method outperforms strong baselines such as D-IM, PV-N, and R&C, particularly in challenging conditions like high occlusion and clutter.