[Paper Review] Universal Humanoid Motion Representations for Physics-Based Control

Zhengyi Luo, Jinkun Cao+5 more|arXiv (Cornell University)|Oct 6, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper introduces PULSE, a universal motion representation for physics-based humanoid control that distills diverse motor skills from a pre-trained motion imitator into a probabilistic latent space using a variational information bottleneck. By jointly learning a proprioception-conditioned prior, PULSE enables efficient, stable, and natural motion generation, significantly improving sample efficiency and performance in hierarchical reinforcement learning across diverse tasks such as terrain traversal and VR motion tracking.

ABSTRACT

We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.

Motivation & Objective

• Address the limited coverage of existing motion representation spaces, which are typically trained on narrow, specialized datasets and fail to generalize to complex or free-form motion tasks.
• Overcome the inefficiency and instability of direct kinematic control in reinforcement learning by learning a compact, expressive latent space for hierarchical control.
• Enable downstream tasks—such as motion tracking with VR controllers and complex terrain traversal—without retraining from scratch by reusing a single, unified motion prior.
• Improve sampling efficiency and exploration in RL by learning a structured, proprioception-conditioned prior that generates diverse, physically plausible motions.
• Demonstrate that a single motion representation can support both generative tasks and estimation tasks (e.g., motion tracking) with high fidelity and stability.

Proposed method

• Train a motion imitator on a large, unstructured motion dataset (e.g., AMASS) to learn high-fidelity human motion from kinematic sequences.
• Distill motor skills directly from the imitator into a low-dimensional latent space using a variational autoencoder with a variational information bottleneck to ensure probabilistic modeling.
• Design an encoder-decoder architecture where the decoder maps latent codes to joint trajectories, and the encoder maps observed motion to latent codes.
• Jointly train a learnable prior conditioned on proprioception (current pose and velocity) to guide sampling and improve exploration in downstream RL.
• Use the prior as a residual action in hierarchical RL, allowing the high-level policy to sample from a structured, physically plausible motion space.
• Apply a regularization loss ($\mathcal{L}_{\text{regu}}$) to penalize large deviations between consecutive latent codes, ensuring continuity and compactness of the latent space.

Experimental results

Research questions

• RQ1 Can a single, unified motion representation be learned that covers a broad spectrum of human motor skills beyond narrow domains like locomotion or dancing?
• RQ2 How can a physics-based motion representation be made both expressive and efficient for downstream reinforcement learning, especially in tasks requiring free-form or complex motion?
• RQ3 What role does a proprioception-conditioned prior play in improving sampling efficiency and stability during hierarchical RL?
• RQ4 To what extent does online distillation from a pre-trained imitator enable coverage of diverse motion styles without requiring task-specific fine-tuning?
• RQ5 Can the learned latent space support both generative tasks (e.g., terrain traversal) and estimation tasks (e.g., VR controller motion tracking) with high performance and realism?

Key findings

• PULSE achieves a 93.4% success rate on the challenging VR controller tracking task when using the full framework (learnable prior, regularization, and joint training), significantly outperforming ablations without these components.
• The use of a regularization term ($\mathcal{L}_{\text{regu}}$) reduces the average global MPJPE error to 88.6 mm and improves success rate by 46.5 percentage points compared to the baseline without regularization.
• The learned prior enables a 71.0% success rate in motion tracking when combined with residual action, compared to only 18.1% when using a fixed Gaussian prior, demonstrating its critical role in exploration.
• PULSE generates more natural and diverse motion than baselines like ASE, CALM, and HuMoR, with fewer implausible motions, as validated by qualitative and quantitative comparisons.
• Training with the full PULSE framework reduces the average MPJPE error to 67.7 mm and the velocity error to 14.9, showing high fidelity in motion tracking.
• The method enables faster convergence in hierarchical RL: policies trained using the PULSE latent space converge faster than training from scratch, as shown by the success rate curve in Figure 5.