[Paper Review] Human Motion Transfer with 3D Constraints and Detail Enhancement

Yang-Tian Sun, Qiancheng Fu+5 more|arXiv (Cornell University)|Mar 30, 2020

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a GAN-based human motion transfer method that uses 3D constraints from reconstructed parametric human models and a detail enhancement network to generate realistic, high-fidelity motion transfer videos. By leveraging Laplacian features for 3D structural consistency and transferring fine details from real source frames, the approach achieves superior qualitative and quantitative results over state-of-the-art methods.

ABSTRACT

We propose a new method for realistic human motion transfer using a generative adversarial network (GAN), which generates a motion video of a target character imitating actions of a source character, while maintaining high authenticity of the generated results. We tackle the problem by decoupling and recombining the posture information and appearance information of both the source and target characters. The innovation of our approach lies in the use of the projection of a reconstructed 3D human model as the condition of GAN to better maintain the structural integrity of transfer results in different poses. We further introduce a detail enhancement net to enhance the details of transfer results by exploiting the details in real source frames. Extensive experiments show that our approach yields better results both qualitatively and quantitatively than the state-of-the-art methods.

Motivation & Objective

• To address the challenge of realistic human motion transfer from source to target subjects while preserving appearance and motion authenticity.
• To overcome artifacts in pose transfer, especially self-occlusion and structural distortion, by incorporating 3D geometric constraints.
• To enhance fine-grained visual details—particularly in hands and facial features—by leveraging geometry from real source frames.
• To improve temporal coherence and image quality in motion transfer by integrating 3D body structure and detail-aware refinement.

Proposed method

• Reconstruct 3D parametric human models from monocular source and target video sequences using a non-rigid 3D reconstruction pipeline.
• Extract Laplacian features from the reconstructed 3D mesh as a deformation-invariant, intrinsic geometric representation for 3D constraint in GAN conditioning.
• Use the 2D pose and projected Laplacian features as conditional input to a GAN-based image-to-image translation network to maintain structural integrity across poses.
• Design a detail enhancement network (DE-Net) that aligns source frame geometry with target poses and transfers high-frequency details (e.g., hands, face) to enhance realism.
• Train the motion transfer network (MT-Net) and DE-Net jointly using both source and target video frames, enabling detail-aware generation.
• Apply an alignment transformation in DE-Net to ensure spatial correspondence between source and target, reducing artifacts in detail transfer.

Experimental results

Research questions

• RQ1 Can 3D geometric constraints derived from reconstructed human models improve the structural fidelity of motion transfer results compared to 2D pose-only conditioning?
• RQ2 How does the use of intrinsic 3D features like Laplacian coordinates compare to traditional 3D features (e.g., UV coordinates) in motion transfer tasks?
• RQ3 To what extent can real source frame details enhance the visual quality of generated motion transfer videos, especially in regions like hands and facial features?
• RQ4 How does the number of source frames during training affect the quality and generalization of the motion transfer model?
• RQ5 What are the failure modes of the method when source subjects exhibit abnormal movements or clothing perturbations?

Key findings

• The proposed method achieves an inception score (IS) of 4.015 and Fréchet Inception Distance (FID) of 51.26, significantly outperforming prior methods with IS 3.612 and FID 58.42.
• Ablation studies confirm that 3D constraints using Laplacian features lead to higher IS and lower FID, demonstrating improved image quality and diversity.
• The detail enhancement network effectively improves visual clarity and reduces blur, particularly in facial features and hands, as shown in qualitative comparisons.
• The method maintains temporal stability and structural consistency across poses, avoiding artifacts like limb mixing common in baseline approaches.
• Experiments show that training convergence in image quality occurs when the source-to-target frame ratio reaches approximately 0.5, providing a data preparation guideline.
• Failure cases reveal limitations when source subjects have long hair or loose clothing, where DE-Net fails to suppress unwanted visual artifacts.