[Paper Review] FLAME: Free-form Language-based Motion Synthesis & Editing

Ji‐Hoon Kim, Jiseob Kim+1 more|arXiv (Cornell University)|Sep 1, 2022

Human Motion and Animation4 citations

TL;DR

FLAME is a diffusion-based generative model for free-form text-to-motion synthesis and editing, leveraging a novel transformer decoder architecture to handle variable-length motion sequences and cross-attention with pre-trained language models. It achieves state-of-the-art performance on HumanML3D, BABEL, and KIT datasets, enabling high-fidelity, diverse motion generation and editing without fine-tuning.

ABSTRACT

Text-based motion generation models are drawing a surge of interest for their potential for automating the motion-making process in the game, animation, or robot industries. In this paper, we propose a diffusion-based motion synthesis and editing model named FLAME. Inspired by the recent successes in diffusion models, we integrate diffusion-based generative models into the motion domain. FLAME can generate high-fidelity motions well aligned with the given text. Also, it can edit the parts of the motion, both frame-wise and joint-wise, without any fine-tuning. FLAME involves a new transformer-based architecture we devise to better handle motion data, which is found to be crucial to manage variable-length motions and well attend to free-form text. In experiments, we show that FLAME achieves state-of-the-art generation performances on three text-motion datasets: HumanML3D, BABEL, and KIT. We also demonstrate that editing capability of FLAME can be extended to other tasks such as motion prediction or motion in-betweening, which have been previously covered by dedicated models.

Motivation & Objective

• To develop a unified framework for text-based 3D motion synthesis and editing that supports free-form language descriptions.
• To address the limitations of prior models that rely on simple action labels, which restrict diversity and controllability.
• To enable flexible conditional generation—such as editing, prediction, and in-betweening—without model fine-tuning.
• To adapt diffusion models to the motion domain, which is inherently spatio-temporal and variable-length.
• To improve sample quality and diversity in motion generation using classifier-free guidance and a tailored transformer architecture.

Proposed method

• Proposes a diffusion-based generative framework for motion data, training the model to denoise motion from isotropic Gaussian noise over 1,000 time steps with cosine beta scheduling.
• Introduces a transformer decoder-based architecture with self-attention, cross-attention, and motion length tokens to model variable-length sequences and attend to free-form text.
• Uses a pre-trained language model (RoBERTa) to encode text prompts, with the CLS token and first 20 tokens fed via cross-attention to guide motion generation.
• Employs classifier-free guidance during inference with 25% of text inputs replaced by empty strings, enabling control over generation quality and alignment.
• Introduces a motion length token to explicitly condition the model on the desired output sequence length, improving generation consistency.
• Freezes the language model during training to stabilize representations and improve performance, validated through ablation studies.

Experimental results

Research questions

• RQ1 Can a diffusion-based model generate high-fidelity, diverse 3D motions that are well-aligned with free-form text descriptions?
• RQ2 Can the same model perform both motion synthesis and editing without fine-tuning, using only textual prompts?
• RQ3 How does the proposed transformer architecture with motion length tokens and cross-attention improve generation quality and controllability?
• RQ4 Can the model generalize to other motion tasks such as prediction and in-betweening through editing?
• RQ5 What is the trade-off between sampling speed and motion quality in diffusion-based motion generation?

Key findings

• FLAME achieves state-of-the-art performance on HumanML3D, BABEL, and KIT datasets across multiple metrics, including FID, FVD, and R-precision.
• On the KIT dataset, FLAME outperforms prior methods when evaluated under the same pipeline used by TEMOS, demonstrating strong generalization.
• The model generates diverse motions with high joint variance (measured across 10 samples per prompt), indicating strong multimodality and expressiveness.
• Generated motions achieve high mCLIP scores (averaged across three trials), confirming strong alignment with text prompts.
• Ablation studies show that freezing the PLM and using cross-attention with full PLM output significantly improve performance.
• Despite slow sampling (30+ seconds per sample at 1,000 steps), reducing diffusion steps to 50–100 maintains high quality, enabling faster inference with minimal loss in fidelity.