[Paper Review] CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds

Ge Gao, Mikko Lauri+3 more|arXiv (Cornell University)|Mar 2, 2021

Robot Manipulation and Learning35 references4 citations

TL;DR

This paper proposes CloudAAE, a 6D object pose estimation system that regresses pose from depth point clouds using a lightweight, on-line data synthesis pipeline and an augmented autoencoder (AAE) to learn a pose-invariant latent code. The method achieves state-of-the-art performance among synthetic-train-only methods on LineMOD, with training data generation up to 1,000x faster than RGB-based PBR methods.

ABSTRACT

It is often desired to train 6D pose estimation systems on synthetic data because manual annotation is expensive. However, due to the large domain gap between the synthetic and real images, synthesizing color images is expensive. In contrast, this domain gap is considerably smaller and easier to fill for depth information. In this work, we present a system that regresses 6D object pose from depth information represented by point clouds, and a lightweight data synthesis pipeline that creates synthetic point cloud segments for training. We use an augmented autoencoder (AAE) for learning a latent code that encodes 6D object pose information for pose regression. The data synthesis pipeline only requires texture-less 3D object models and desired viewpoints, and it is cheap in terms of both time and hardware storage. Our data synthesis process is up to three orders of magnitude faster than commonly applied approaches that render RGB image data. We show the effectiveness of our system on the LineMOD, LineMOD Occlusion, and YCB Video datasets. The implementation of our system is available at: https://github.com/GeeeG/CloudAAE.

Motivation & Objective

• To address the high cost and domain gap of RGB-based synthetic data for 6D pose estimation.
• To reduce training data generation cost while maintaining high pose estimation accuracy using depth-only point clouds.
• To develop a lightweight, on-line data synthesis pipeline that requires only texture-less 3D models and desired viewpoints.
• To enable accurate 6D pose regression using a latent code from an AAE that encodes only pose-invariant features.
• To demonstrate state-of-the-art performance using only synthetic data on benchmark datasets like LineMOD.

Proposed method

• The system uses a point cloud-based augmented autoencoder (AAE) to learn a latent code that encodes 6D object pose while being invariant to non-pose-related variations via input augmentation.
• The AAE reconstructs noise- and occlusion-free point cloud segments in the desired 6D pose, ensuring the latent code captures only pose-relevant information.
• The latent code is fed into two separate deep networks to regress 3D rotation and 3D translation independently.
• The on-line data synthesis pipeline generates synthetic point cloud segments using only texture-less 3D models and desired viewpoints, avoiding expensive PBR rendering.
• Synthetic data is generated in under 30ms per 128 segments on a single GPU, with only 128MB storage for 13 objects and 100k poses.
• The system uses standard ICP refinement post-prediction to improve accuracy, with an additional 0.02s inference cost.

Figure 1: Left, Data synthesis pipeline. The input for our on-line training pipeline is a 3D object model $P^{O}$ with class information, and the desired 3D rotation $R$ and translation $\mathbf{t}$ . $P^{O}$ is at object coordinate $O$ . With $R$ and $\mathbf{t}$ , the object model is first transfo

Experimental results

Research questions

• RQ1 Can a lightweight, on-line point cloud synthesis pipeline replace costly PBR-based RGB rendering for 6D pose estimation?
• RQ2 Does an AAE-based latent code effectively disentangle pose information from other variations in point clouds?
• RQ3 Can a 6D pose estimator trained exclusively on synthetic depth point clouds achieve state-of-the-art performance on real-world benchmarks?
• RQ4 How does the amount of synthetic training data and latent code dimensionality affect generalization and accuracy?
• RQ5 To what extent does the domain gap between synthetic and real data impact performance, and can it be mitigated with depth-only synthesis?

Key findings

• On the LineMOD dataset, the proposed method achieves 82.1% average ADD-S accuracy using only 100k synthetic samples per object, outperforming other synthetic-only methods.
• With 100k samples per class, the model achieves near-optimal performance; increasing to 1M samples provides no further gain, indicating data efficiency.
• The best generalization occurs with a 1024-dimensional latent code, where both rotation and translation regression perform optimally.
• The system's synthetic data generation is up to 1,000x faster than PBR-based RGB rendering, with only 128MB storage for 13 objects and 100k poses.
• On YCB-Video, the method achieves comparable performance to RGBD-based methods (e.g., DenseFusion, PVN3D) when using real data with synthetic data from the proposed pipeline.
• The T-SNE visualization confirms that the learned latent codes are well-clustered per object class, indicating effective pose encoding.

Figure 2: Point cloud based Augmented Autoencoder. The numbers of neurons in EC-MLP and MLP layers are indicated in parentheses. EC is short for EdgeConv. Dimensions of intermediate features are indicated without parentheses. Skip connections denote the concatenation of edge features. For the encode