[Paper Review] AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

Ruilong Li, Shan Yang+2 more|arXiv (Cornell University)|Jan 21, 2021

Human Motion and AnimationEngineering81 references32 citations

TL;DR

Introduces AIST++, a large multi-modal dataset of 3D dance motion with music, and FACT, a Full-Attention Cross-modal Transformer that generates long, music-conditioned 3D dance sequences from a 2-second seed motion.

ABSTRACT

We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses -- the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict $N$ future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively.

Motivation & Objective

• Motivate the need for realistic music-conditioned 3D dance generation and provide a large-scale dataset to study this task.
• Develop a model that can generate long sequences of 3D dance motion aligned with input music from a short seed sequence.
• Demonstrate that full-attention cross-modal fusion with future-N supervision yields stable, high-quality motion generation.
• Enable easy retargeting of generated motion to novel characters through SMPL-based representations.

Proposed method

• Propose AIST++: a dataset with 5.2 hours of 3D dance motion, 1408 sequences, 10 genres, acquired from multi-view videos with known camera poses.
• Introduce FACT: a deep cross-modal transformer with three components—an audio transformer, a seed-motion transformer, and a cross-modal transformer with full-attention that fuses both modalities early.
• Train FACT to predict N future motion frames (N = 20 in experiments) in an auto-regressive setup, using future-N supervision to stabilize long-range generation.
• Represent 3D dance as joint rotations and global translation to support retargeting to new characters.
• Use early fusion of audio and motion embeddings and a 12-layer cross-modal transformer to learn music-motion correlations effectively.

Figure 2: Cross-Modal Music Conditioned 3D Motion Generation Overview. Our proposed a Full-Attention Cross-modal Transformer (FACT) network (details in Figure 3 ) takes in a music piece and a $2$ -second sequence of seed motion, then auto-regressively generates long-range future motions that correla

Experimental results

Research questions

• RQ1 Can a large-scale, multi-view dataset like AIST++ enable robust learning of music-conditioned 3D dance generation?
• RQ2 Does a full-attention cross-modal transformer with future-N supervision outperform causal-attention and shallow fusion baselines in producing long, music-consistent dance sequences?
• RQ3 How does early fusion of audio and motion modalities affect the model’s ability to follow music cues?
• RQ4 What metrics best capture motion quality, diversity, and music-dance alignment in generated sequences?

Key findings

• AIST++ is validated as a large, multi-view, multi-genre dataset with 1408 sequences, 30 subjects, 10 genres, and synchronized multi-view images.
• FACT with full-attention, future-N supervision, and early cross-modal fusion produces longer, non-freezing, and more music-aligned 3D dance motion than baselines.
• FACT achieves better motion realism (lower FID_k and FID_g) and higher BeatAlign scores than state-of-the-art methods Li et al. 2021, Dancenet, and DanceRevolution on AIST++.
• A user study shows FACT-generated motions are perceived as more music-coherent than the baselines in 81% of comparisons vs Li et al., 71% vs Dancenet, and 77% vs DanceRevolution.
• Ablation studies confirm the necessity of full-attention with future-N supervision and early cross-modal fusion for high-quality, music-conditioned generation.

Figure 3: FACT Model Details. (a) The structure of the audio/motion/cross-modal transformer with $N$ attention layers. (b) Attention and supervision mechanism as a simplified two-layer model. Models like GPT [ 66 ] and the motion generator of [ 55 ] use causal attention (left) to predict the immedia