[Paper Review] EDGE: Editable Dance Generation From Music

Jonathan Tseng, Rodrigo Castellon+1 more|arXiv (Cornell University)|Nov 19, 2022

Music Technology and Sound Studies4 citations

TL;DR

EDGE is a diffusion-based generative model for editable, physically plausible dance motion conditioned on music, leveraging Jukebox for audio features and a transformer architecture to enable joint-wise conditioning and in-betweening. It achieves state-of-the-art performance through a novel Physical Foot Contact Score and user studies validating superior quality over prior methods.

ABSTRACT

Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods. Qualitative samples from our model can be found at our website.

Motivation & Objective

• To develop a method for generating realistic, long-form dance motions that are physically plausible and synchronized with music.
• To address limitations in existing evaluation metrics by proposing a new, physically grounded metric for foot contact consistency.
• To enable powerful editing capabilities—such as joint-wise conditioning and in-betweening—within a unified generative framework.
• To improve dance generation quality beyond prior state-of-the-art methods through both quantitative metrics and large-scale human evaluation.

Proposed method

• A transformer-based diffusion model is trained to denoise latent dance sequences from random noise, conditioned on music embeddings.
• Jukebox, a pre-trained music generation model, provides high-level audio representations as cross-attention context for the diffusion process.
• A novel Contact Consistency Loss is introduced to reduce foot-sliding by enforcing consistent foot contact behavior across time steps.
• The Physical Foot Contact Score (PFC) is proposed as an acceleration-based, physics-inspired metric to evaluate physical plausibility without explicit dynamics simulation.
• The model supports editable generation via joint-wise conditioning and in-betweening, enabling user-specified motion constraints.
• Long sequences are generated by chaining locally consistent, shorter clips, enabling arbitrary-length output.

Figure 2 : EDGE Pipeline Overview: EDGE learns to denoise dance sequences from time $t=T$ to $t=0$ , conditioned on music. Music embedding information is provided by a frozen Jukebox model [ 5 ] and acts as cross-attention context. EDGE takes a noisy sequence $\bm{z}_{T}\sim\mathcal{N}(0,\bm{I})$ an

Experimental results

Research questions

• RQ1 Can a diffusion-based model generate long, physically plausible, and musically aligned dance sequences with strong editability?
• RQ2 Do existing automated metrics for dance generation accurately reflect human-evaluated quality?
• RQ3 Can a learned metric like PFC effectively capture physical plausibility in kinematic motion without explicit physics simulation?
• RQ4 How does the integration of Jukebox features improve dance generation quality compared to hand-crafted audio features?
• RQ5 To what extent do editing capabilities like joint-wise conditioning and in-betweening enhance user control and motion quality?

Key findings

• EDGE achieves state-of-the-art performance on the AIST++ dataset, significantly outperforming prior methods in user studies.
• The Physical Foot Contact Score (PFC) effectively identifies and penalizes foot-sliding, improving physical plausibility without requiring physics-based simulation.
• User studies show that EDGE-generated dances are rated as more realistic and musically aligned than those from previous state-of-the-art models.
• The beat alignment metric used in prior work was found to be flawed, as it penalizes valid non-beat-aligned motions such as double-time or half-time choreographies.
• The PFC metric correlates better with human evaluation than traditional FID-style metrics, which were shown to be unreliable for motion generation.
• EDGE supports flexible editing, including joint-wise conditioning and in-betweening, enabling precise user control over generated dance sequences.

Figure 3 : Although EDGE is trained on 5-second clips, it can generate choreographies of any length by imposing temporal constraints on batches of sequences. In this example, EDGE constrains the first 2.5 seconds of each sequence to match the last 2.5 seconds of the previous one to generate a 12.5-s