[Paper Review] Learning human behaviors from motion capture by adversarial imitation

Josh Merel, Yuval Tassa+6 more|arXiv (Cornell University)|Jul 7, 2017

Reinforcement Learning in RoboticsComputer Science1 references154 citations

TL;DR

The paper extends generative adversarial imitation learning (GAIL) to train humanlike motion policies from motion capture with partial observations, enabling body transfer and reuse as sub-skills within higher-level controllers.

ABSTRACT

Rapid progress in deep reinforcement learning has made it increasingly feasible to train controllers for high-dimensional humanoid bodies. However, methods that use pure reinforcement learning with simple reward functions tend to produce non-humanlike and overly stereotyped movement behaviors. In this work, we extend generative adversarial imitation learning to enable training of generic neural network policies to produce humanlike movement patterns from limited demonstrations consisting only of partially observed state features, without access to actions, even when the demonstrations come from a body with different and unknown physical parameters. We leverage this approach to build sub-skill policies from motion capture data and show that they can be reused to solve tasks when controlled by a higher level controller.

Motivation & Objective

• Motivate and enable humane, humanlike movement generation for high-dimensional humanoids without hand-engineered rewards.
• Develop an imitation learning pipeline that works with partial state features and without action data.
• Demonstrate body transfer, robustness to noisy motion capture, and reuse of learned sub-skills in higher-level control.
• Showcase the emergence of robust transitions between multiple behaviors through multi-behavior learning.

Proposed method

• Extend GAIL to partial observations and include a context variable for multi-behavior policy learning.
• Train a stochastic neural policy to output Gaussian action distributions for actuators.
• Use TRPO for policy updates and adversarial rewards derived from a discriminator distinguishing demo vs. policy data.
• Provide end-to-end pipeline from motion capture-based demonstrations to low-level controllers, then integrate with a high-level controller for task learning.
• Utilize a MuJoCo physics engine with varied bodies, including a complex humanoid, for training and evaluation.
• Expose end-effector–based features (vectors from root to feet, hands, head) and inertial sensors to stabilize imitation from noisy motion captures.

Experimental results

Research questions

• RQ1 Can GAIL immitation learning succeed when demonstrations have only partial state observations and no actions?
• RQ2 Is imitation robust to differences in body dynamics between demonstrator and imitator (body transfer)?
• RQ3 Can multiple behaviors be learned and transitioned between robustly using a context-conditioned discriminator?
• RQ4 Can learned low-level skills from motion capture be reused by a higher-level controller to solve new tasks?
• RQ5 How well can a complex humanoid learn from limited, noisy motion capture data and still exhibit humanlike motions?

Key findings

• Imitation learning with partial state observations can match behaviors without access to actions.
• Discriminator conditioning on body-invariant features enables cross-body imitation and re-targeting across different dof configurations.
• Multi-behavior training with context variables yields robust transitions between skills and supports switching during trajectories.
• Motion-capture based learning yields more natural gait and capable get-up behaviors for a complex humanoid than random initialization or pure RL.
• End-effector–based feature representations stabilize imitation from noisy motion capture and improve perceived humanlikeness of learned motions.
• Sub-skills learned from motion capture can be composed and modulated by a higher-level controller to perform tasks like navigation, turning, running, and stair climbing.