[Paper Review] Learning Motion Priors for 4D Human Body Capture in 3D Scenes

Siwei Zhang, Yan Zhang+3 more|arXiv (Cornell University)|Aug 23, 2021

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes LEMO, a method that learns motion priors from the AMASS dataset to improve 4D human body capture from monocular RGB(D) videos in complex 3D scenes. By combining a data-driven motion smoothness prior and a contact-aware motion infiller trained via per-instance self-supervised learning, LEMO reduces jitter and foot-skating artifacts, achieving smoother, more natural, and occlusion-robust motion reconstructions than prior methods.

ABSTRACT

Recovering high-quality 3D human motion in complex scenes from monocular videos is important for many applications, ranging from AR/VR to robotics. However, capturing realistic human-scene interactions, while dealing with occlusions and partial views, is challenging; current approaches are still far from achieving compelling results. We address this problem by proposing LEMO: LEarning human MOtion priors for 4D human body capture. By leveraging the large-scale motion capture dataset AMASS, we introduce a novel motion smoothness prior, which strongly reduces the jitters exhibited by poses recovered over a sequence. Furthermore, to handle contacts and occlusions occurring frequently in body-scene interactions, we design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. To prove the effectiveness of the proposed motion priors, we combine them into a novel pipeline for 4D human body capture in 3D scenes. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions. The code and data are available at https://sanweiliti.github.io/LEMO/LEMO.html.

Motivation & Objective

• To address the challenge of reconstructing high-quality, natural human motion from monocular RGB(D) sequences in complex 3D scenes, where occlusions and partial views degrade reconstruction quality.
• To reduce motion jitter and foot-skating artifacts common in existing monocular capture pipelines like PROX.
• To improve the realism and physical plausibility of human-scene interactions in 4D reconstruction by modeling contact dynamics and full-body motion jointly.
• To develop a self-supervised, per-instance fine-tuning strategy that adapts a general motion infiller to specific test instances with partial observations.
• To close the performance gap between monocular RGB(D) capture and high-end marker-based motion capture systems using data-driven priors.

Proposed method

• A fully convolutional autoencoder learns a motion smoothness prior in a latent space that captures holistic full-body dynamics over a large spatio-temporal receptive field.
• Body motion is represented using surface markers instead of joints, enabling better modeling of limb rotation and degrees of freedom.
• A contact-aware motion infiller is trained on AMASS to jointly predict body motion and foot-ground contact status, reducing foot-skating artifacts.
• A per-instance self-supervised fine-tuning scheme adapts the pre-trained infiller to test sequences by minimizing a loss on visible body parts only.
• The method integrates a physics-inspired contact friction term to further regularize foot-ground interactions and improve physical plausibility.
• A multi-stage optimization pipeline combines the motion priors, contact modeling, and SMPL-X fitting to reconstruct smooth, accurate, and occlusion-robust 4D human motions.

Experimental results

Research questions

• RQ1 Can a data-driven motion smoothness prior learned from large-scale mocap data significantly reduce jitter and improve temporal coherence in monocular 4D human motion reconstruction?
• RQ2 Can a contact-aware motion infiller trained on AMASS effectively reconstruct occluded body parts while preserving realistic foot-ground interactions and reducing skating artifacts?
• RQ3 Does per-instance self-supervised fine-tuning of the motion infiller lead to better adaptation to individual test sequences with partial observations compared to a fixed prior?
• RQ4 How does the integration of motion priors and contact modeling improve the realism and physical plausibility of human-scene interactions in complex 3D scenes?
• RQ5 Can the proposed pipeline achieve motion reconstruction quality comparable to marker-based mocap systems using only monocular RGB(D) input?

Key findings

• The proposed motion smoothness prior reduces motion jitter and improves reconstruction quality on both the PROX and 3DPW datasets, with measurable gains in MPJPE, MPMPE, and PVE metrics.
• The contact-aware motion infiller outperforms a heuristic contact rule, reducing foot-skating ratio and achieving lower reconstruction errors for occluded body parts (MPJPE-L and MPMPE-L).
• Self-supervised per-instance fine-tuning consistently improves performance on both full-body and occluded-body metrics, demonstrating effective adaptation to individual test instances.
• When integrated into the pipeline, the motion infiller significantly improves 2D joint accuracy and reduces PSKL and non-collision scores on the PROX dataset compared to baseline methods.
• Swapping the order of stages shows that the infiller fails when applied to jittered PROX outputs, confirming that motion smoothing is essential before infilling.
• Ablation studies confirm that both the contact-aware infiller and self-supervised fine-tuning are critical components, with performance degrading when either is removed.