[Paper Review] Self-supervised Learning of Motion Capture

Hsiao-Yu Fish Tung, Hsiao-Wei Tung+2 more|arXiv (Cornell University)|Dec 4, 2017

Advanced Vision and ImagingComputer Science31 references131 citations

TL;DR

The paper presents a learning-based motion capture model for monocular videos that is pretrained on synthetic data and refined at test time via self-supervised, differentiable rendering losses for keypoints, segmentation, and dense mesh motion, outperforming traditional optimization and non-adaptive baselines.

ABSTRACT

Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.

Motivation & Objective

• Motivate monocular 3D motion capture without clean backgrounds or multi-camera setups.
• Develop a neural model that predicts SMPL 3D human mesh parameters from monocular video.
• Leverage synthetic data for supervision and self-supervision via differentiable rendering to adapt at test time.
• Demonstrate that test-time self-supervision yields tighter 3D reconstructions than purely supervised or purely optimization-based approaches.

Proposed method

• Use SMPL as the dense 3D human mesh model with parameters theta (pose) and beta (shape).
• Pretrain the network on synthetic data (Surreal) with supervised theta and beta regression.
• Apply end-to-end self-supervised losses by differentiably rendering 3D keypoints, dense mesh motion, and segmentation, then compare with detected 2D counterparts.
• Self-supervised losses include keypoint reprojection, motion reprojection against 2D optical flow, and segmentation reprojection via Chamfer-distance-based penalties.
• Implement visibility via ray casting to mask motion reprojection for occluded vertices, and train with back-propagation.
• Evaluate on Surreal and Human3.6M (H3.6M), comparing against optimization-based baselines and pretrained-only models.

Experimental results

Research questions

• RQ1 Can a neural network learn to predict SMPL parameters from monocular video when trained with synthetic data and adapted through self-supervision at test time?
• RQ2 Do differentiable rendering-based losses (keypoint, motion, segmentation) enable accurate 3D reconstruction and domain transfer from synthetic to real data?
• RQ3 Is test-time adaptation essential to surpass purely pretrained or purely optimization-based approaches in monocular motion capture?
• RQ4 How do the proposed self-supervised losses complement each other in improving 3D mesh and skeleton accuracy?

Key findings

• The self-supervised, test-time adaptation yields higher 3D reconstruction accuracy than pretrained-only or direct optimization baselines.
• On Surreal, the pretrained+Self-Sup model achieves surface error 74.5 mm, per-joint error 64.4 mm, and reconstruction error 203.9 mm, outperforming baselines.
• On H3.6M, the pretrained+Self-Sup model reduces per-joint error to 98.4 mm and reconstruction error to 145.8 mm, versus optimization and pretrained baselines.
• Ablation shows all three losses (keypoint, segmentation, motion) are complementary and collectively improve 3D keypoint and mesh accuracy.
• Self-supervision via differentiable rendering enables domain transfer from synthetic to real data (Surreal to H3.6M) with improved fitting.
• The approach combines supervised pretraining with unsupervised adaptation, achieving tighter mesh fits without manual initialization.