[Paper Review] Motion Flow Matching for Human Motion Synthesis and Editing

Vincent Tao Hu, Wenzhe Yin+8 more|arXiv (Cornell University)|Dec 14, 2023

Human Motion and Animation4 citations

TL;DR

This paper introduces Motion Flow Matching, a novel generative model for human motion synthesis that achieves state-of-the-art performance with only ten sampling steps—dramatically reducing inference time compared to prior diffusion models—while enabling efficient, training-free motion editing via sampling trajectory rewriting. The method leverages ODE-based flow matching to ensure straight generation trajectories, enabling fast, high-fidelity motion generation and consistent editing across diverse tasks like in-betweening, prediction, and upper-body manipulation.

ABSTRACT

Human motion synthesis is a fundamental task in computer animation. Recent methods based on diffusion models or GPT structure demonstrate commendable performance but exhibit drawbacks in terms of slow sampling speeds and error accumulation. In this paper, we propose \emph{Motion Flow Matching}, a novel generative model designed for human motion generation featuring efficient sampling and effectiveness in motion editing applications. Our method reduces the sampling complexity from thousand steps in previous diffusion models to just ten steps, while achieving comparable performance in text-to-motion and action-to-motion generation benchmarks. Noticeably, our approach establishes a new state-of-the-art Fréchet Inception Distance on the KIT-ML dataset. What is more, we tailor a straightforward motion editing paradigm named \emph{sampling trajectory rewriting} leveraging the ODE-style generative models and apply it to various editing scenarios including motion prediction, motion in-between prediction, motion interpolation, and upper-body editing. Our code will be released.

Motivation & Objective

• To address the slow sampling speed of diffusion-based human motion generation models, which typically require thousands of steps.
• To improve controllability and editability in motion generation without requiring fine-tuning or additional training.
• To explore the application of flow matching—a technique from other modalities—within human motion generation for the first time.
• To enable practical, real-time motion editing by leveraging the straight trajectory property of flow matching models.
• To establish a new SOTA in Fréchet Inception Distance (FID) on the KIT-ML dataset with minimal sampling steps.

Proposed method

• Proposes a flow matching-based generative model for human motion, trained to regress a linearly interpolated vector field via an ODE solver.
• Uses a Transformer-based architecture to model motion sequences, enabling efficient and scalable generation.
• Employs sampling trajectory rewriting: a training-free editing method that aligns known motion segments with the ODE trajectory to generate consistent, edited motions.
• Applies the ODE solver to generate motion from noise to data by following a straight path, ensuring stability and fast convergence.
• Performs editing by rewriting the sampling trajectory from a known time step (e.g., t=0.2) onward, preserving consistency with input motion segments.
• Conducts inpainting in joint space to enable semantic editing of specific body parts without affecting others.

Experimental results

Research questions

• RQ1 Can flow matching be effectively applied to human motion generation to achieve high-quality results with significantly fewer sampling steps than diffusion models?
• RQ2 How can the straight trajectory property of flow matching be exploited to enable efficient, training-free motion editing?
• RQ3 Can sampling trajectory rewriting maintain motion consistency when editing partial sequences, such as in in-betweening or future prediction?
• RQ4 Does the proposed method outperform existing diffusion-based models in both generation quality and inference speed on standard benchmarks?
• RQ5 Can the model generalize across diverse editing scenarios, including upper-body manipulation and motion interpolation, while preserving semantic coherence?

Key findings

• The proposed Motion Flow Matching model achieves a new state-of-the-art Fréchet Inception Distance (FID) of 15.4 on the KIT-ML dataset, outperforming prior methods.
• The model reduces sampling steps from ~1000 in diffusion models to just 10, achieving a 100x speedup over MDM and 5x speedup over MLD on a V100 GPU.
• Inference time is reduced to an average of 0.11 seconds per motion on the HumanML3D test set, demonstrating real-time feasibility.
• Sampling trajectory rewriting enables consistent editing across multiple scenarios, including in-betweening, future prediction, upper-body editing, and interpolation, with slight improvements in FID, ADE, and FDE over MDM.
• Even at early time steps (e.g., t=0.2), the model produces reasonably accurate motion estimations that align closely with the prompt, demonstrating the stability of straight trajectories.
• The method maintains high performance in text-to-motion and action-to-motion generation, with comparable or better results than baselines while using far fewer function evaluations.