[Paper Review] Generative Tweening: Long-term Inbetweening of 3D Human Motions

Yi Zhou, Jingwan Lu+4 more|arXiv (Cornell University)|May 18, 2020

Human Motion and AnimationEngineering53 references20 citations

TL;DR

This paper introduces a conditional generative adversarial network for long-term inbetweening of 3D human motions, using a two-stage approach that first predicts local joint rotations and then infers global character trajectories, while enforcing biomechanical constraints via a range-constrained forward kinematics layer. The method enables diverse, realistic motion generation from sparse keyframes using a novel 'Motion DNA' concept for style control, achieving real-time synthesis of complex, natural-looking motions with high user control.

ABSTRACT

The ability to generate complex and realistic human body animations at scale, while following specific artistic constraints, has been a fundamental goal for the game and animation industry for decades. Popular techniques include key-framing, physics-based simulation, and database methods via motion graphs. Recently, motion generators based on deep learning have been introduced. Although these learning models can automatically generate highly intricate stylized motions of arbitrary length, they still lack user control. To this end, we introduce the problem of long-term inbetweening, which involves automatically synthesizing complex motions over a long time interval given very sparse keyframes by users. We identify a number of challenges related to this problem, including maintaining biomechanical and keyframe constraints, preserving natural motions, and designing the entire motion sequence holistically while considering all constraints. We introduce a biomechanically constrained generative adversarial network that performs long-term inbetweening of human motions, conditioned on keyframe constraints. This network uses a novel two-stage approach where it first predicts local motion in the form of joint angles, and then predicts global motion, i.e. the global path that the character follows. Since there are typically a number of possible motions that could satisfy the given user constraints, we also enable our network to generate a variety of outputs with a scheme that we call Motion DNA. This approach allows the user to manipulate and influence the output content by feeding seed motions (DNA) to the network. Trained with 79 classes of captured motion data, our network performs robustly on a variety of highly complex motion styles.

Motivation & Objective

• To address the challenge of generating long, complex, and realistic 3D human motions from very sparse user-specified keyframes.
• To maintain biomechanical plausibility and keyframe accuracy over extended motion sequences.
• To enable user control over motion style through a novel 'Motion DNA' mechanism that conditions the generator on seed motions.
• To support flexible input formats, including partial body poses and 2D sketches, for greater authoring flexibility.
• To achieve real-time or near real-time motion synthesis for interactive animation workflows.

Proposed method

• A two-stage conditional GAN architecture: first generating local joint rotations, then predicting global motion trajectories based on those rotations.
• Incorporation of a Range-Constrained Forward Kinematics (RC-FK) layer that enforces biomechanical limits on joint rotations using Euler angle representations with fixed, discontinuity-free orders and ranges.
• Use of a holistic, context-aware generation strategy that considers all keyframes simultaneously, rather than interpolating between adjacent pairs.
• Introduction of 'Motion DNA'—a latent conditioning scheme that allows the network to generate multiple diverse motion sequences from the same keyframe constraints by feeding in different seed motion patterns.
• Adaptation of the input representation to support partial body inputs (e.g., only root, half-body, or 2D projections), with masked joint inputs during training and inference.
• Training on 79 classes of captured motion data from the CMU Mocap dataset to generalize across diverse motion styles.

Experimental results

Research questions

• RQ1 How can a deep generative model produce long, natural, and biomechanically plausible 3D human motions from very sparse keyframes?
• RQ2 How can the model ensure accurate keyframe pose adherence while maintaining global motion coherence over extended sequences?
• RQ3 How can the model generate multiple distinct, plausible motion sequences under the same keyframe constraints?
• RQ4 Can the model generalize to partial or low-fidelity inputs such as 2D sketches or root-only positions?
• RQ5 Can the model achieve real-time performance for interactive animation and iterative design?

Key findings

• The model generates highly realistic and diverse 3D human motion sequences from as few as four sparse keyframes, with natural poses and meaningful motion variations.
• The two-stage architecture—first local motion, then global path prediction—enables accurate long-term motion synthesis while maintaining biomechanical plausibility.
• The RC-FK layer successfully prevents unrealistic joint rotations by constraining Euler angle ranges and orders, reducing discontinuities in the motion space.
• The Motion DNA mechanism allows users to influence the style and variation of output motions by conditioning on different seed motions, enabling multiple plausible results per keyframe set.
• The system supports flexible input formats, including partial body poses and 2D sketches, and can synthesize plausible 3D motions even from non-uniform or distorted 2D inputs.
• The method achieves real-time performance, generating one minute of motion in under half a second, with real-time updates possible for shorter sequences.