[Paper Review] Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Ye Yuan, Kris Kitani|arXiv (Cornell University)|Jun 12, 2020

Human Pose and Action RecognitionComputer Science58 references34 citations

TL;DR

RFC augments humanoid control with learnable residual forces to overcome dynamics mismatch, enabling agile motion imitation (e.g., ballet) and multi-modal long-term motion via a dual-policy framework.

ABSTRACT

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jeté. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at https://www.ye-yuan.com/rfc.

Motivation & Objective

• Address dynamics mismatch between humanoid models and real humans to improve motion imitation.
• Enable highly agile motions (e.g., ballet) that prior physics-based methods struggle with.
• Develop a dual-policy framework for multi-modal, long-term motion synthesis without task guidance or user input.
• Leverage large-scale motion datasets (Human3.6M) to learn diverse long-horizon motions.

Proposed method

• Introduce residual force control (RFC) that augments a humanoid policy with external residual forces in the action space.
• Formulate RFC as RFC-Explicit (explicit residual forces with contact points) or RFC-Implicit (total residual joint torques) and couple with a composite policy.
• Model dynamics with an augmented equation of motion including residual terms (Eq. 2 for explicit, Eq. 4 for implicit).
• Regularize residual forces via a reward term to keep modified dynamics close to original physics (Eq. 3, Eq. 5).
• Use a dual-policy control framework: a kinematic policy (CVAE) forecast futures and an RFC-based policy imitates those futures for physically plausible motion.
• Train using PPO within a physics simulator (MuJoCo), with PD controllers and reference motions from motion capture.

Experimental results

Research questions

• RQ1 Can residual forces compensate for dynamics mismatch to enable imitation of highly agile motions?
• RQ2 How do RFC-Explicit and RFC-Implicit compare in learning efficiency and motion quality?
• RQ3 Can a dual-policy framework generate multi-modal long-term motions without task guidance or user input?
• RQ4 Does learning from large motion datasets (e.g., Human3.6M) enable diverse long-horizon motion synthesis?

Key findings

• RFC converges faster and yields higher-quality motions than the state-of-the-art DeepMimic on agile motions.
• RFC enables highly agile ballet moves such as pirouette, arabesque, and jeté in simulation.
• RFC-Explicit and RFC-Implicit perform comparably in imitation performance, with RFC-Implicit offering computational efficiency advantages.
• A dual-policy framework allows synthesizing stable, multi-modal long-term motions without task guidance or user input.
• The approach learns from Human3.6M and can produce diverse long-term motions beyond short reference clips.