[Paper Review] Human Motion Diffusion as a Generative Prior

Yonatan Shafir, Guy Tevet+2 more|arXiv (Cornell University)|Mar 2, 2023

Human Motion and AnimationEngineering33 citations

TL;DR

The paper introduces three diffusion-prior based motion composition methods—sequential, parallel, and model composition (DiffusionBlending)—to enable long-horizon, multi-person, and controllable human motion generation using a pretrained Motion Diffusion Model (MDM).

ABSTRACT

Recent work has demonstrated the significant potential of denoising diffusion models for generating human motion, including text-to-motion capabilities. However, these methods are restricted by the paucity of annotated motion data, a focus on single-person motions, and a lack of detailed control. In this paper, we introduce three forms of composition based on diffusion priors: sequential, parallel, and model composition. Using sequential composition, we tackle the challenge of long sequence generation. We introduce DoubleTake, an inference-time method with which we generate long animations consisting of sequences of prompted intervals and their transitions, using a prior trained only for short clips. Using parallel composition, we show promising steps toward two-person generation. Beginning with two fixed priors as well as a few two-person training examples, we learn a slim communication block, ComMDM, to coordinate interaction between the two resulting motions. Lastly, using model composition, we first train individual priors to complete motions that realize a prescribed motion for a given joint. We then introduce DiffusionBlending, an interpolation mechanism to effectively blend several such models to enable flexible and efficient fine-grained joint and trajectory-level control and editing. We evaluate the composition methods using an off-the-shelf motion diffusion model, and further compare the results to dedicated models trained for these specific tasks.

Motivation & Objective

• Address data limitations in human motion by leveraging a pretrained diffusion prior (MDM) for new composition tasks.
• Enable long-sequence generation via sequential composition (DoubleTake) without retraining on long data.
• Achieve few-shot two-person motion generation by learning a slim communicator (ComMDM) between fixed priors.
• Provide flexible, fine-grained control through model composition (DiffusionBlending) and targeted fine-tuning.

Proposed method

• Use a fixed pretrained Motion Diffusion Model (MDM) as a prior for new tasks.
• Propose DoubleTake for long sequences: two-stage inference with handshakes to couple adjacent intervals and refine transitions.
• Introduce ComMDM, a slim communication block that coordinates two fixed priors for two-person motion in a few-shot setting.
• Fine-tune MDM for trajectory and joint control by masking control features during diffusion to enforce adherence (single-control fine-tuning).
• Introduce DiffusionBlending to compose multiple conditioned models via a generalized classifier-free guidance for cross-joint control.

Figure 1. We suggest three novel motion composition methods, all based on the recent Motion Diffusion Model (MDM). (Left) Sequential composition generating an arbitrary long motion with text control over each time interval. (Middle) Parallel composition generating two-person motion from text. A diff

Experimental results

Research questions

• RQ1 Can a pretrained motion diffusion prior be repurposed to generate arbitrarily long motions without retraining on long data?
• RQ2 Is it feasible to generate convincing two-person interactions with only a few training examples by adding a coordination module between fixed priors?
• RQ3 How can diffusion-based controllers be blended or fine-tuned to achieve fine-grained, joint-level control over motion trajectories?
• RQ4 Do model composition techniques yield superior or comparable results to dedicated models for specific motion tasks?

Key findings

• DoubleTake enables 10-minute long fluent motions by composing short-clip priors and providing per-interval control.
• ComMDM can coordinate two fixed priors to produce two-person motions with few-shot training and outperforms baselines in prefix completion and text-guided generation (user studies).
• Fine-tuned control and DiffusionBlending enable cross-combinations of control signals (e.g., root and hand) and improve alignment with target trajectories and joints.
• Across benchmarks (BABEL, HumanML3D, 3DPW), the proposed methods either outperform or closely approach dedicated task-specific models in multiple metrics (R-precision, FID, diversity, etc.).
• The approach demonstrates zero-shot or few-shot viability for long-horizon, multi-person, and controllable motion generation using diffusion priors.

Figure 2. Soft blending overview. We allow b frames long linear masking between $\mathbf{M_{hard}}$ to $\mathbf{M_{soft}}$ such that during the Second take at every denoising step part of the originally generated motion (suffix or prefix) going through refinement to fit the transition.