[Paper Review] Dance Generation with Style Embedding: Learning and Transferring Latent Representations of Dance Styles

Xinjian Zhang, Yi Xu+3 more|arXiv (Cornell University)|Apr 30, 2021

Music and Audio Processing21 references4 citations

TL;DR

This paper proposes a frame-by-frame music-to-dance generation framework with controllable style embeddings that learn latent style representations from reference dance clips and transfer them to generate diverse, style-consistent dances. By using a transformer-based music encoder and a style-aware dance generator, the model achieves superior beat matching, emotion alignment, style consistency, and diversity compared to baselines, validated on a newly constructed large-scale music-to-dance dataset with anime, popping, and locking styles.

ABSTRACT

Choreography refers to creation of dance steps and motions for dances according to the latent knowledge in human mind, where the created dance motions are in general style-specific and consistent. So far, such latent style-specific knowledge about dance styles cannot be represented explicitly in human language and has not yet been learned in previous works on music-to-dance generation tasks. In this paper, we propose a novel music-to-dance synthesis framework with controllable style embeddings. These embeddings are learned representations of style-consistent kinematic abstraction of reference dance clips, which act as controllable factors to impose style constraints on dance generation in a latent manner. Thus, the dance styles can be transferred to dance motions by merely modifying the style embeddings. To support this study, we build a large music-to-dance dataset. The qualitative and quantitative evaluations demonstrate the advantage of our proposed framework, as well as the ability of synthesizing diverse styles of dances from identical music via style embeddings.

Motivation & Objective

• To address the lack of explicit modeling of style-specific knowledge in music-to-dance generation, which limits diversity and realism.
• To enable controllable style transfer in dance generation by learning latent style representations from reference dance clips.
• To develop a framework that generates diverse, style-consistent dances from the same music by modifying only the style embedding.
• To construct a large-scale, style-annotated music-to-dance dataset for training and evaluation of style-aware generation models.
• To demonstrate that frame-level alignment between music and dance, combined with style embeddings, improves both style fidelity and motion diversity.

Proposed method

• A style embedding producer learns latent style representations from reference dance clips using a prototype-based method in a shared latent space.
• The music-to-dance generator uses a transformer encoder to extract temporal and semantic features from music input.
• The dance generator combines music representations and learned style embeddings to generate motion sequences frame-by-frame.
• Style embeddings are computed as learnable weighted combinations of prototype vectors, enabling flexible style transfer.
• The framework employs frame-aligned generation to ensure temporal synchronization between music beats and dance motions.
• A realistic video rendering model is used to visualize generated dances for qualitative evaluation.

Experimental results

Research questions

• RQ1 Can latent style representations be effectively learned and transferred to generate diverse, style-consistent dances from the same music?
• RQ2 How does the proposed framework compare to existing methods in beat matching, emotion alignment, and style consistency?
• RQ3 To what extent can style embeddings enable controllable style transfer without retraining?
• RQ4 Does the use of frame-level music-dance alignment improve generation quality compared to global music representation?
• RQ5 How does the model perform in generating diverse dance motions when conditioned on different initial poses and style embeddings?

Key findings

• The proposed framework achieves a 1.3% higher beat hit rate than Dancing2Music, demonstrating improved temporal alignment between music and dance.
• The model exhibits strong emotion matching, with generated dance intensity curves closely following the input music's intensity profile.
• The FID score for style consistency is lower than baselines, indicating higher similarity between generated and real dance styles.
• The diversity score—measured as FID between style embeddings of generated dances with different initial poses—shows the model is significantly more diverse than Dancing2Music.
• User studies confirm that the model outperforms baselines in style consistency, with participants preferring its outputs for style accuracy.
• Despite strong performance, generated dances still show limb distortions in some cases, indicating room for improvement in motion realism.