[Paper Review] Human Motion Diffusion Model

Guy Tevet, Sigal Raab+4 more|arXiv (Cornell University)|Sep 29, 2022

Human Pose and Action Recognition166 citations

TL;DR

MDM uses a transformer-based, classifier-free diffusion model that predicts the motion sample x0 directly, enabling lightweight training with geometric losses; it achieves state-of-the-art results on text-to-motion and action-to-motion benchmarks and supports editing and in-betweening.

ABSTRACT

Natural and expressive human motion generation is the holy grail of computer animation. It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain. MDM is transformer-based, combining insights from motion generation literature. A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion step. This facilitates the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss. As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion and action-to-motion. https://guytevet.github.io/mdm-page/ .

Motivation & Objective

• Motivate a lightweight yet expressive diffusion approach for human motion generation.
• Leverage geometric losses (position, foot contact, velocity) to improve motion realism.
• Enable multiple conditioning modes (text-to-motion, action-to-motion, unconditioned) with classifier-free guidance.
• Demonstrate editing and in-betweening capabilities via diffusion-based inpainting in motion data.
• Show practical training efficiency (≈3 days on a mid-range GPU) and competitive benchmarks.

Proposed method

• Adopt a transformer-encoder backbone that processes motion sequences ( joints x D ).
• Predict the clean motion x0 at each denoising step t, instead of predicting the noise, following a simple loss L_simple=E[||x0−G(xt,t,c)||^2].
• Incorporate geometric losses: L_pos to align predicted positions, L_foot to reduce foot sliding, and L_vel to match velocities.
• Train with classifier-free guidance by randomly dropping conditioning c (≈10% of samples) to learn p(x0|c) and enable sampling with guidance scale s.
• Condition on CLIP-based text embeddings for text-to-motion or learn action embeddings for action-to-motion; support unconditioned generation (c=empty).
• Use diffusion inpainting for editing: fix parts of the motion and generate missing segments or re-synthesize body parts during sampling.

Experimental results

Research questions

• RQ1 Can a lightweight, transformer-based diffusion model accurately capture the many-to-many nature of text-to-motion and action-to-motion tasks?
• RQ2 Do geometric losses tailored to motion (positions, foot contacts, velocities) improve diffusion-based motion quality and realism?
• RQ3 Is classifier-free guidance effective for balancing fidelity and diversity in motion generation across multiple conditioning modalities?
• RQ4 Can diffusion-based editing and in-betweening be achieved without retraining, using motion inpainting in joint space?
• RQ5 What are the practical training and inference requirements to reach state-of-the-art results on standard benchmarks?

Key findings

• MDM achieves state-of-the-art results on text-to-motion benchmarks HumanML3D and KIT.
• User studies show evaluators preferred MDM over comparable methods in many cases, with 42.3% preferred over ground truth in one test.
• MDM outperforms prior methods on action-to-motion benchmarks HumanAct12 and UESTC across FID, diversity, and multimodality metrics (foot-contact loss improves results).
• Diffusion with a transformer backbone trains on ~3 days on a single RTX 2080 Ti, with ≈1000 noising steps and a cosine schedule.
• Editing and in-betweening are achievable by diffusion inpainting in both temporal and spatial domains, enabling motion completion and body-part edits without retraining.