[Paper Review] Human Motion Modeling using DVGANs

Xiao Lin, Mohamed R. Amer|arXiv (Cornell University)|Apr 27, 2018

Human Pose and Action Recognition32 references42 citations

TL;DR

DVGANs present a dense-validation Wasserstein GAN framework for human motion generation and completion, using both CNN and RNN generators and a dense, multi-resolution discriminator conditioned on text descriptions. The model achieves long, diverse, and realistic motion sequences and benchmarks on CMU Mocap and Human3.6M with inception scores and retrieval metrics.

ABSTRACT

We present a novel generative model for human motion modeling using Generative Adversarial Networks (GANs). We formulate the GAN discriminator using dense validation at each time-scale and perturb the discriminator input to make it translation invariant. Our model is capable of motion generation and completion. We show through our evaluations the resiliency to noise, generalization over actions, and generation of long diverse sequences. We evaluate our approach on Human 3.6M and CMU motion capture datasets using inception scores.

Motivation & Objective

• Motivate and address the challenge of generating long, diverse human motion from scratch and for motion completion.
• Propose a DVGANs architecture that combines CNN/RNN generators with a dense, multi-resolution CNN discriminator.
• Condition motion generation on textual descriptions to enable zero-shot or unseen actions.
• Show robustness to noise and ability to generate long sequences beyond short-term predictions.
• Benchmark generation quality and retrieval alignment on large motion capture datasets.

Proposed method

• Introduce DVGANs, combining CNN and RNN components for generator and discriminator.
• Use Wasserstein GAN with Gradient Penalty (WGAN-GP) for stable training.
• Condition both generator and discriminator on text descriptions rather than fixed class labels.
• Implement dense validation at multiple time-resolutions within the discriminator to better detect artifacts.
• Incorporate data augmentation by translating input sequences to enforce translation-invariance.
• Provide two generator variants: an RNN-based generator for generation and completion, and a CNN-based generator for generation via progressive upsampling of low-frame-rate animations.
• Formulate a CNN-based discriminator that encodes videos and text and uses multi-resolution validation scores.
• Train using L2-regularized optimization with Lipschitz constraints, and apply a final-cut mechanism to produce fixed-length outputs.

Experimental results

Research questions

• RQ1 Can DVGANs generate long, diverse, realistic human motion sequences from text descriptions without seed frames?
• RQ2 How does dense, multi-resolution validation in the discriminator affect generation quality and stability?
• RQ3 Does conditioning on text enable generalization to unseen actions and improve motion completion?
• RQ4 What is the impact of data augmentation and final-cut sampling on generation and retrieval metrics?
• RQ5 How do CNN and RNN generator architectures compare for motion generation and completion tasks?

Key findings

• DVGANs achieve long, high-quality motion generation and can perform motion completion.
• Dense validation across time-resolutions improves inception scores and retrieval metrics compared to single-layer validation.
• Data augmentation and final-cut strategies provide additional gains, with notable improvements in retrieval accuracy.
• CNN-based generator generally yields higher inception scores and strong generation capabilities, while RNN-based generator excels at motion completion.
• Experiments on CMU Mocap and Human3.6M demonstrate realistic, diverse motions and the model’s ability to generalize across actions and even unseen tasks.