[Paper Review] Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

Zimo Li, Yi Zhou+4 more|arXiv (Cornell University)|Jul 17, 2017

Human Pose and Action RecognitionComputer Science150 citations

TL;DR

The paper proposes auto-conditioned RNNs (acRNN) for long, diverse, and complex human motion synthesis, mitigating error accumulation to generate hundreds of seconds of motion.

ABSTRACT

We present a real-time method for synthesizing highly complex human motions using a novel training regime we call the auto-conditioned Recurrent Neural Network (acRNN). Recently, researchers have attempted to synthesize new motion by using autoregressive techniques, but existing methods tend to freeze or diverge after a couple of seconds due to an accumulation of errors that are fed back into the network. Furthermore, such methods have only been shown to be reliable for relatively simple human motions, such as walking or running. In contrast, our approach can synthesize arbitrary motions with highly complex styles, including dances or martial arts in addition to locomotion. The acRNN is able to accomplish this by explicitly accommodating for autoregressive noise accumulation during training. Our work is the first to our knowledge that demonstrates the ability to generate over 18,000 continuous frames (300 seconds) of new complex human motion w.r.t. different styles.

Motivation & Objective

• Motivate and enable generation of highly complex human motions beyond walking/running.
• Address error accumulation in autoregressive motion generation.
• Demonstrate long-horizon synthesis (hundreds of seconds) across multiple styles.

Proposed method

• Introduce acRNN where the network is trained using its own past outputs as inputs (auto-conditioning).
• Use condition length u and generation length v to feed v predicted frames during training alongside ground-truth frames.
• Represent motion data as relative joint displacements (root motion and joint positions) to capture consistent periodicity.
• Train acLSTM with three fully connected layers (memory size 1024) using ADAM on sequences of length 100 over 500k iterations.
• Evaluate against baselines (LSTM, ERD, seq2seq, scheduled sampling) on CMU motion capture subsets with Euclidean loss.

Experimental results

Research questions

• RQ1 Can acRNN generate arbitrarily long, realistic, and style-varied motions (e.g., dances, martial arts) without freezing?
• RQ2 How does auto-conditioning affect long-range motion stability and realism compared to standard RNN training?
• RQ3 What are the trade-offs of different condition lengths on short-term prediction error and long-term motion continuity?
• RQ4 How does acRNN perform across different motion styles relative to existing baselines?

Key findings

• acLSTM generates motion without freezing for hundreds of seconds (e.g., 300+ seconds shown in results).
• acLSTM achieves lower short- and mid-term prediction errors than several baselines for multiple styles, notably Indian dance and martial arts.
• Vanilla LSTM tends to freeze after ~60 frames, while acLSTM maintains ongoing varied motion.
• Compared to ERD and seq2seq, acLSTM provides longer-lasting plausible motion, though all methods show increasing error for very long horizons due to human motion stochasticity.
• Mixing training data from different styles yields hybrid motions that borrow characteristics from both styles.
• Motion sequences can be generated at ~60 fps, with examples demonstrating 50-second + synthesized outputs.