[Paper Review] CPS++: Improving Class-level 6D Pose and Shape Estimation From Monocular Images With Self-Supervised Learning

Fabian Manhardt, Gu Wang+6 more|arXiv (Cornell University)|Mar 12, 2020

Robotics and Sensor-Based Localization70 references35 citations

TL;DR

CPS++ introduces class-level monocular 6D pose and metric shape estimation with an end-to-end differentiable pipeline, incorporating a self-supervised extension to bridge synthetic-to-real domain gaps. It achieves state-of-the-art pose accuracy and provides a learnable 3D shape representation for each object class.

ABSTRACT

Contemporary monocular 6D pose estimation methods can only cope with a handful of object instances. This naturally hampers possible applications as, for instance, robots seamlessly integrated in everyday processes necessarily require the ability to work with hundreds of different objects. To tackle this problem of immanent practical relevance, we propose a novel method for class-level monocular 6D pose estimation, coupled with metric shape retrieval. Unfortunately, acquiring adequate annotations is very time-consuming and labor intensive. This is especially true for class-level 6D pose estimation, as one is required to create a highly detailed reconstruction for all objects and then annotate each object and scene using these models. To overcome this shortcoming, we additionally propose the idea of synthetic-to-real domain transfer for class-level 6D poses by means of self-supervised learning, which removes the burden of collecting numerous manual annotations. In essence, after training our proposed method fully supervised with synthetic data, we leverage recent advances in differentiable rendering to self-supervise the model with unannotated real RGB-D data to improve latter inference. We experimentally demonstrate that we can retrieve precise 6D poses and metric shapes from a single RGB image.

Motivation & Objective

• Motivate the need for class-level monocular 6D pose estimation that scales beyond instance-specific models.
• Propose CPS to jointly estimate 6D pose and metric shape from a single RGB image.
• Introduce AtlasNet-based per-class shape latent spaces to reconstruct object shapes.
• Enable self-supervised (synthetic-to-real) domain transfer to reduce annotation burdens.

Proposed method

• Use RetinaNet to detect 2D regions of interest and RoIAlign to extract features for each detection.
• For each detection, predict allocentric rotation q_a, 2D image centroid, depth z, metric size (w,h,l), and a 32-d shape latent e per class.
• Represent shape with AtlasNet encoders/decoders trained per class, and predict a shape offset from the class mean latent shape m_c.
• Back-project to 3D to obtain 3D pose and use a differentiable 3D alignment loss based on Chamfer distance between predicted and ground-truth point clouds.
• Introduce a 3D point-cloud alignment loss that jointly optimizes pose and shape parameters in 3D space.
• Implement a self-supervised extension (CPS++) that renders differentiable RGB-D pairs from predicted meshes and enforces geometric and mask-based alignment with real unlabeled data.
• Train with synthetic data supervision plus Self6D-inspired self-supervision to bridge domain gaps.

Experimental results

Research questions

• RQ1 Can a monocular network estimate 6D pose and metric shape for class-level object categories unseen during training?
• RQ2 Does end-to-end training with a differentiable 3D alignment loss improve pose accuracy and shape quality?
• RQ3 Can self-supervised learning using real unlabeled RGB-D data bridge the synthetic-to-real domain gap for class-level 6D pose estimation?
• RQ4 Is it feasible to learn per-class 3D shape latents that generalize across instance variations within a class?

Key findings

• Proposes a novel CPS framework that jointly predicts 6D pose, object scale, and a class-specific shape latent, enabling 3D shape reconstruction from a single RGB image.
• Introduces a differentiable 3D alignment loss that improves pose accuracy by directly optimizing the alignment in 3D space.
• Demonstrates a self-supervised extension CPS++ that leverages real unlabeled RGB-D data to close the synthetic-to-real gap.
• Shows that per-class AtlasNet-based shape spaces enable metric shape estimation and can be regularized to stay within learned shape distributions.
• Collects and releases over 30k real RGB-D samples to facilitate self-supervised learning in class-level 6D pose estimation.