[Paper Review] 3D Human Motion Estimation via Motion Compression and Refinement

Zhengyi Luo, S. Alireza Golestaneh+1 more|arXiv (Cornell University)|Aug 9, 2020

Human Pose and Action Recognition51 references10 citations

TL;DR

This paper proposes MEVA, a two-stage 3D human motion estimation method that first compresses video sequences into a smooth, general motion representation using a Variational Autoencoder (VAE), then refines it with a person-specific regressor to recover fine-grained motion details. The approach reduces acceleration error by 54.3% on 3DPW while achieving state-of-the-art MPJPE, enabling both accurate and temporally smooth 3D pose sequences.

ABSTRACT

We develop a technique for generating smooth and accurate 3D human pose and motion estimates from RGB video sequences. Our method, which we call Motion Estimation via Variational Autoencoder (MEVA), decomposes a temporal sequence of human motion into a smooth motion representation using auto-encoder-based motion compression and a residual representation learned through motion refinement. This two-step encoding of human motion captures human motion in two stages: a general human motion estimation step that captures the coarse overall motion, and a residual estimation that adds back person-specific motion details. Experiments show that our method produces both smooth and accurate 3D human pose and motion estimates.

Motivation & Objective

• To address the lack of temporal smoothness in state-of-the-art 3D human motion estimation, despite high per-joint accuracy.
• To decompose motion estimation into coarse (general) and fine (person-specific) components for improved plausibility and accuracy.
• To learn a disentangled motion representation that separates smooth, shared human motion from individual motion variations.
• To reduce jitter and unnatural motion artifacts in 3D pose sequences without sacrificing spatial accuracy.

Proposed method

• A Variational Autoencoder (VAE) is trained on the AMASS dataset to learn a low-dimensional latent space representing smooth, general human motion.
• The VAE is fine-tuned with data augmentation—random root rotation, frame rate variation, and left-right flipping—to improve robustness to unseen motion variations.
• A coarse motion estimate is generated by decoding the latent code from the VAE, representing the overall smooth motion trajectory.
• A Motion Refinement Regressor (MRR) is trained to predict residual offsets from the coarse estimate using image evidence from video frames.
• The final 3D pose sequence is obtained by adding the MRR-predicted residuals to the VAE-generated coarse motion.
• The entire framework is trained in a two-stage manner: first pretraining the VAE, then jointly training the MRR on top of the frozen VAE.

Experimental results

Research questions

• RQ1 Can a two-stage motion estimation framework improve temporal smoothness without degrading spatial accuracy in 3D human pose estimation?
• RQ2 How effective is a VAE-based motion compression step in capturing the intrinsic smoothness of general human motion?
• RQ3 To what extent can a person-specific refinement module recover motion details not captured by a general motion model?
• RQ4 Does using a pretrained VAE latent space improve generalization and reduce kinematically invalid poses compared to end-to-end training?

Key findings

• MEVA reduces acceleration error by 54.3% on the 3DPW dataset compared to prior state-of-the-art methods, significantly improving motion smoothness.
• The ablation study shows that using a pretrained VAE leads to more stable and plausible motion sequences, avoiding kinematically invalid poses seen in end-to-end training.
• Training only the Variational Motion Estimator (VME) achieves performance comparable to HMMR, a prior SOTA method for low acceleration error.
• The combination of random root rotation, frame rate variation, and left-right flipping during VAE pretraining reduces VAE reconstruction error by up to 70% on unseen 3DPW sequences.
• MEVA achieves a PA-MPJPE of 54.7 on 3DPW, outperforming prior methods in both accuracy and smoothness.
• Visualizations confirm that the coarse motion captures global motion trends, while the residual refinement adds back dynamic, person-specific details like sudden arm swings or head turns.