[Paper Review] Occluded Human Body Capture with Self-Supervised Spatial-Temporal Motion Prior

Buzhen Huang, Yuan Shu+2 more|arXiv (Cornell University)|Jul 12, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a self-supervised spatial-temporal motion prior learned from non-occluded data to improve monocular 3D human motion capture under occlusion. By encoding 2D motions into maps and synthesizing occlusions, the method trains a joint-level spatial-temporal network using masked image modeling, achieving accurate, coherent 3D motion recovery from real occluded videos with strong generalization and efficiency.

ABSTRACT

Although significant progress has been achieved on monocular maker-less human motion capture in recent years, it is still hard for state-of-the-art methods to obtain satisfactory results in occlusion scenarios. There are two main reasons: the one is that the occluded motion capture is inherently ambiguous as various 3D poses can map to the same 2D observations, which always results in an unreliable estimation. The other is that no sufficient occluded human data can be used for training a robust model. To address the obstacles, our key-idea is to employ non-occluded human data to learn a joint-level spatial-temporal motion prior for occluded human with a self-supervised strategy. To further reduce the gap between synthetic and real occlusion data, we build the first 3D occluded motion dataset~(OcMotion), which can be used for both training and testing. We encode the motions in 2D maps and synthesize occlusions on non-occluded data for the self-supervised training. A spatial-temporal layer is then designed to learn joint-level correlations. The learned prior reduces the ambiguities of occlusions and is robust to diverse occlusion types, which is then adopted to assist the occluded human motion capture. Experimental results show that our method can generate accurate and coherent human motions from occluded videos with good generalization ability and runtime efficiency. The dataset and code are publicly available at \url{https://github.com/boycehbz/CHOMP}.

Motivation & Objective

• Address the inherent ambiguity in 3D human motion capture from occluded monocular videos, where multiple 3D poses can correspond to the same 2D observations.
• Overcome the lack of real-world occluded 3D motion data for training robust models by creating a new synthetic-to-real generalizable benchmark.
• Improve motion capture accuracy and temporal coherence in occlusion scenarios by leveraging joint-level spatial-temporal correlations learned from non-occluded data.
• Develop a self-supervised learning strategy to learn a motion prior that reduces ambiguity without requiring real occlusion annotations.
• Enable efficient inference by training a lightweight network that leverages temporal and kinematic priors for real-time performance.

Proposed method

• Represent 2D human motion as motion maps to encode spatial and temporal information jointly, enabling consistent occlusion synthesis on non-occluded data.
• Synthesize realistic occlusions on non-occluded 2D motion maps to simulate diverse occlusion types during self-supervised pre-training.
• Design a self-supervised masked image modeling (MIM) task to reconstruct full motion maps from occluded inputs, learning a robust motion prior.
• Introduce a spatial-temporal layer combining dilated convolutions and vision transformer blocks to model joint-level spatial correlations and global motion dependencies.
• Train the lifting network with the self-supervised prior to guide 3D motion estimation from visible 2D joints in real occlusion scenarios.
• Use state-of-the-art 2D pose detectors to extract visible joint coordinates in inference, feeding them into the model to predict full 3D motion.

Experimental results

Research questions

• RQ1 Can a self-supervised prior learned from non-occluded data effectively reduce ambiguity in 3D human motion estimation under occlusion?
• RQ2 How does joint-level spatial-temporal modeling improve motion coherence and accuracy in occluded scenarios compared to purely temporal models?
• RQ3 To what extent does the proposed method generalize across diverse occlusion types and proportions without real occlusion annotations?
• RQ4 Can synthetic occlusions on non-occluded data effectively bridge the domain gap to real-world occlusion data?
• RQ5 How does the integration of a self-supervised prior improve runtime efficiency and prediction quality compared to end-to-end optimization-based methods?

Key findings

• The proposed method achieves a mean MPJPE of 58.2 mm on the OcMotion dataset, significantly outperforming the baseline temporal model (70.5 mm) and ablation variants.
• With the addition of the self-supervised prior and spatial-temporal layer, the model reduces MPJPE by 12.3% and PVE by 12.1% compared to the baseline.
• The method demonstrates robustness to occlusion ratios from 0% to 50%, with minimal performance degradation across varying occlusion proportions.
• The inclusion of the self-supervised prior improves PA-MPJPE from 43.2 mm (without prior) to 36.1 mm, indicating better joint-level accuracy.
• The model achieves inference speed of 36.4 FPS, demonstrating high runtime efficiency suitable for real-time applications.
• Ablation studies confirm that the spatial-temporal layer and self-supervised prior are both essential, with the full model outperforming variants by a large margin.