[Paper Review] Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision

Dushyant Mehta, Helge Rhodin+4 more|arXiv (Cornell University)|Nov 29, 2016

Human Pose and Action Recognition34 citations

TL;DR

This paper proposes a novel CNN-based method for monocular 3D human pose estimation that leverages improved CNN supervision through skeletal kinematic relationships and multi-level skip connections, combined with transfer learning from 2D pose models, achieving over 25% improvement on standard benchmarks. It introduces a new diverse, markerless motion-capture-based benchmark for real-world generalization.

ABSTRACT

We propose a new CNN-based method for regressing 3D human body pose from a single image that improves over the state-of-the-art on standard benchmarks by more than 25%. Our approach addresses the limited generalizability of models trained solely on the starkly limited publicly available 3D body pose data. Improved CNN supervision leverages first and second order parent relationships along the skeletal kinematic tree, and improved multi-level skip connections to learn better representations through implicit modification of the loss landscape. Further, transfer learning from 2D human pose prediction significantly improves accuracy and generalizability to unseen poses and camera views. Additionally, we contribute a new benchmark and training set for human body pose estimation from monocular images of real humans, that has ground truth captured with marker-less motion capture. It complements existing corpora with greater diversity in pose, human appearance, clothing, occlusion, and viewpoints, and enables increased scope of augmentation. The benchmark covers outdoors and indoor scenes.

Motivation & Objective

• To address the limited generalizability of 3D human pose models trained on scarce public 3D pose data.
• To improve feature representation learning through enhanced CNN supervision using skeletal kinematic relationships.
• To enhance model generalization to unseen poses and camera views via transfer learning from 2D pose prediction.
• To create a new, diverse benchmark with real-world variations in pose, appearance, clothing, occlusion, and viewpoints for monocular 3D pose estimation.
• To support broader data augmentation and more robust evaluation using markerless motion-captured ground truth.

Proposed method

• The method employs first and second-order parent relationships in the skeletal kinematic tree to improve CNN supervision.
• It uses improved multi-level skip connections to modify the loss landscape and learn better hierarchical representations.
• Transfer learning is applied by initializing the model with weights from a pre-trained 2D human pose estimation network.
• The network is trained end-to-end using a regression loss on 3D joint coordinates with enhanced supervision signals.
• A new benchmark is constructed using markerless motion capture to collect diverse, real-world monocular image sequences with 3D ground truth.
• The training data includes varied poses, human appearances, clothing, occlusions, and indoor/outdoor scenes for robustness.

Experimental results

Research questions

• RQ1 How can improved CNN supervision through skeletal kinematic relationships enhance 3D pose estimation accuracy?
• RQ2 To what extent does transfer learning from 2D pose models improve generalization to unseen poses and camera views?
• RQ3 Can multi-level skip connections effectively modify the loss landscape to improve feature learning in 3D pose regression?
• RQ4 How does a diverse, real-world benchmark with markerless motion capture data impact model performance and evaluation?
• RQ5 What is the performance gain of the proposed method over state-of-the-art on standard benchmarks?

Key findings

• The proposed method improves performance on standard benchmarks by more than 25% compared to the state-of-the-art.
• The integration of skeletal kinematic relationships through first and second-order parent supervision enhances model accuracy.
• Transfer learning from 2D pose prediction significantly improves generalization to unseen poses and camera viewpoints.
• The improved multi-level skip connections lead to better feature representation learning by implicitly modifying the loss landscape.
• The new benchmark with markerless motion-captured data enables more diverse and realistic training and evaluation, supporting broader augmentation and generalization.
• The method demonstrates strong robustness across diverse real-world conditions including occlusion, varied clothing, and complex viewpoints.