[Paper Review] PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

Xiang Yu, Tanner Schmidt+2 more|arXiv (Cornell University)|Nov 1, 2017

Human Pose and Action Recognition34 references130 citations

TL;DR

PoseCNN is a CNN that estimates 6D object pose by decoupling 3D translation (via 2D center localization and center depth) and 3D rotation (via quaternion regression), with ShapeMatch-Loss for symmetry, evaluated on YCB-Video and OccludedLINEMOD datasets.

ABSTRACT

Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at https://rse-lab.cs.washington.edu/projects/posecnn/.

Motivation & Objective

• Motivate robust 6D pose estimation in clutter and occlusion without heavy reliance on depth data.
• Develop an end-to-end CNN that separately handles translation and rotation estimation.
• Handle symmetric objects through a specialized loss function (ShapeMatch-Loss).
• Provide a large-scale RGB-D video dataset (YCB-Video) with 6D pose annotations for 21 objects.

Proposed method

• A two-stage CNN backbone that shares features across tasks.
• Semantic labeling per pixel to identify object classes and enable center voting.
• 2D object center localization via regressing unit center directions for each pixel and a Hough voting layer to find 2D centers.
• 3D translation estimation by combining 2D center location with predicted center distance (depth) to recover T.
• 3D rotation regression from object bounding box features to a per-class quaternion; train with PoseLoss for non-symmetric objects and ShapeMatch-Loss for symmetric objects.
• ICP refinement using depth data to refine poses when available.

Experimental results

Research questions

• RQ1 Can a CNN jointly perform semantic labeling, 2D center voting, and 3D pose regression to achieve accurate 6D pose estimation in cluttered scenes?
• RQ2 How can symmetry be effectively handled during rotation regression without explicitly enumerating symmetric orientations?
• RQ3 Does a center-voting-based translation estimation improve robustness to occlusions compared to direct 3D coordinate regression?
• RQ4 How does PoseCNN perform on color-only versus RGB-D inputs on challenging datasets like OccludedLINEMOD and YCB-Video?

Key findings

• PoseCNN achieves strong 6D pose estimation from color images alone, outperforming a 3D coordinate regression baseline on YCB-Video.
• Incorporating depth via ICP refinement significantly improves accuracy, often surpassing RGB-D baselines.
• ShapeMatch-Loss effectively handles symmetric objects, improving pose estimation for Eggbox and Glue in OccludedLINEMOD.
• On OccludedLINEMOD, PoseCNN with ICP outperforms state-of-the-art methods using RGB-D inputs for several objects.
• The YCB-Video dataset (21 objects, 133,827 frames) provides robust training and evaluation for occlusions and symmetries.