[Paper Review] Dynamic Future Net: Diversified Human Motion Generation

Wenheng Chen, He Wang+3 more|arXiv (Cornell University)|Aug 25, 2020

Human Pose and Action Recognition38 references4 citations

TL;DR

This paper proposes Dynamic Future Net (DFN), a deep generative model that explicitly models short- and long-term motion stochasticity by leveraging bidirectional state representations—past and future—within a latent space. Trained on minimal motion data (as little as 7 seconds), DFN generates diverse, high-quality, and temporally coherent human motion sequences with arbitrary duration, outperforming prior methods in diversity and robustness through distribution-based loss functions.

ABSTRACT

Human motion modelling is crucial in many areas such as computer graphics, vision and virtual reality. Acquiring high-quality skeletal motions is difficult due to the need for specialized equipment and laborious manual post-posting, which necessitates maximizing the use of existing data to synthesize new data. However, it is a challenge due to the intrinsic motion stochasticity of human motion dynamics, manifested in the short and long terms. In the short term, there is strong randomness within a couple frames, e.g. one frame followed by multiple possible frames leading to different motion styles; while in the long term, there are non-deterministic action transitions. In this paper, we present Dynamic Future Net, a new deep learning model where we explicitly focuses on the aforementioned motion stochasticity by constructing a generative model with non-trivial modelling capacity in temporal stochasticity. Given limited amounts of data, our model can generate a large number of high-quality motions with arbitrary duration, and visually-convincing variations in both space and time. We evaluate our model on a wide range of motions and compare it with the state-of-the-art methods. Both qualitative and quantitative results show the superiority of our method, for its robustness, versatility and high-quality.

Motivation & Objective

• To address the challenge of generating diverse, high-quality human motion sequences from limited training data.
• To explicitly model both short-term (frame-level) and long-term (action-level) stochasticity in human motion dynamics.
• To develop a generative model that maintains motion quality while enabling high diversity in generated sequences.
• To enable training on mixed or single motion types without retraining, enhancing versatility.
• To replace point-wise loss with distributional similarity loss for better modeling of motion randomness.

Proposed method

• DFN employs a variational autoencoder to embed motion sequences into a low-dimensional latent space, enabling efficient dynamics modeling.
• It uses a bidirectional recurrent network to jointly model past and future states, capturing both short-term pose transitions and long-term action changes.
• The model explicitly conditions the current state on both historical and future context, enabling dynamic modeling of stochastic transitions.
• A novel distributional loss function is introduced, comparing the predicted and ground-truth pose distributions rather than point estimates, enhancing diversity.
• The architecture supports end-to-end training on mixed motion types, enabling multi-modal dynamics learning from diverse data.
• Inference uses a conditional sampling strategy based on the latent code and future context, allowing arbitrary-length motion generation.

Experimental results

Research questions

• RQ1 Can a deep generative model effectively capture both short-term and long-term stochasticity in human motion dynamics?
• RQ2 How well can a model generate diverse, high-quality motion sequences when trained on minimal data (e.g., <15 seconds)?
• RQ3 Can a single model generalize across multiple motion types (e.g., walking, running, boxing) without retraining?
• RQ4 Does modeling future context improve motion diversity and temporal coherence compared to autoregressive or unidirectional models?
• RQ5 To what extent does a distributional loss function enhance motion diversity compared to standard point-wise reconstruction losses?

Key findings

• DFN successfully generates diverse, visually convincing motion sequences with arbitrary duration, even when trained on as little as 7 seconds of motion data.
• The model maintains high motion quality and temporal coherence, with generated motions showing natural pose variations and plausible action transitions.
• When trained on mixed motion types, DFN captures multi-modal dynamics and generates diverse motion styles from a single prefix, such as walking to running or dancing.
• The distributional loss function leads to significantly more diverse motion outputs compared to point-wise reconstruction, as validated by mean-distance distribution matching to training data.
• Generated motions show strong statistical similarity to the training data in pose distribution, with mean-distance distributions across time steps closely matching real data.
• Even with drastically reduced training data (down to 7 seconds), DFN maintains stable and plausible motion generation, demonstrating robustness to data scarcity.