[Paper Review] Domain Transfer for 3D Pose Estimation from Color Images without Manual Annotations

Mahdi Rad, Markus Oberweger+1 more|arXiv (Cornell University)|Jan 1, 2018

Human Pose and Action Recognition50 references51 citations

TL;DR

The paper presents a method to estimate 3D pose from color images without color annotations by learning a color-to-depth feature mapping using paired RGB-D data and aligning real and synthetic depth domains with Maximum Mean Discrepancy (MMD). It applies to both 3D object pose and 3D hand pose estimation.

ABSTRACT

We introduce a novel learning method for 3D pose estimation from color images. While acquiring annotations for color images is a difficult task, our approach circumvents this problem by learning a mapping from paired color and depth images captured with an RGB-D camera. We jointly learn the pose from synthetic depth images that are easy to generate, and learn to align these synthetic depth images with the real depth images. We show our approach for the task of 3D hand pose estimation and 3D object pose estimation, both from color images only. Our method achieves performances comparable to state-of-the-art methods on popular benchmark datasets, without requiring any annotations for the color images.

Motivation & Objective

• Motivate 3D pose estimation from color images without labor-intensive color image annotations.
• Leverage labeled depth images (real or synthetic) to train a robust pose estimator.
• Bridge the domain gap between color and depth modalities without target-domain annotations.

Proposed method

• Train a depth feature extractor f_D and a pose estimator h_D on synthetic depth data.
• Train a color feature extractor f_C and a mapping network g to map color features to depth features using RGB-D pairs and a feature-mapping loss L_FM.
• Use Maximum Mean Discrepancy (MMD) to align real and synthetic depth feature distributions in RKHS.
• Combine pose loss on synthetic depth data, feature-mapping loss, and MMD loss in a joint objective (with weights beta and gamma).
• Inference maps color features to depth feature space via g∘f_C and applies h_D to estimate 3D pose from color images.

Experimental results

Research questions

• RQ1 Can 3D pose be estimated from color images without any color annotations by leveraging paired RGB-D data and depth-only supervision?
• RQ2 How effectively can a learned color-to-depth feature mapping transfer pose knowledge from synthetic depth data to real color images?
• RQ3 Does MMD-based domain alignment between real and synthetic depth features suffice to bridge the domain gap without real color annotations?
• RQ4 Is the approach applicable to both rigid object pose estimation and 3D hand pose estimation from color images?

Key findings

• Achieves pose estimation performance comparable to state-of-the-art methods on LINEMOD and STB/RHD benchmarks without annotated color images.
• On LINEMOD, surpasses methods that require color annotations and approaches the performance of the best annotation-dependent method.
• For 3D hand pose, competitive results are reported on STB with comparable accuracy to annotation-based methods, using RGB-D pairs for learning.
• Inference times are favorable: ~3.2 ms for object pose and ~8.6 ms for hand pose on a TITAN X platform.
• Training requires synthetic depth data and RGB-D pairs, with pretraining steps improving convergence.