[Paper Review] Music-oriented Dance Video Synthesis with Pose Perceptual Loss

Xuanchi Ren, Haoran Li+2 more|arXiv (Cornell University)|Dec 13, 2019

Human Motion and AnimationEngineering44 references19 citations

TL;DR

This paper proposes a two-stage GAN-based framework for music-oriented dance video synthesis that generates realistic, beat-aligned dance videos from any given music. It introduces a novel pose perceptual loss to train on noisy OpenPose skeletons and employs dual discriminators (local temporal and global content) to ensure motion coherence and rhythm alignment, achieving user-perceived realism comparable to professional choreography.

ABSTRACT

We present a learning-based approach with pose perceptual loss for automatic music video generation. Our method can produce a realistic dance video that conforms to the beats and rhymes of almost any given music. To achieve this, we firstly generate a human skeleton sequence from music and then apply the learned pose-to-appearance mapping to generate the final video. In the stage of generating skeleton sequences, we utilize two discriminators to capture different aspects of the sequence and propose a novel pose perceptual loss to produce natural dances. Besides, we also provide a new cross-modal evaluation to evaluate the dance quality, which is able to estimate the similarity between two modalities of music and dance. Finally, a user study is conducted to demonstrate that dance video synthesized by the presented approach produces surprisingly realistic results. The results are shown in the supplementary video at https://youtu.be/0rMuFMZa_K4

Motivation & Objective

• To generate photo-realistic, music-aligned dance videos from arbitrary music inputs, enabling personalized content creation.
• To address the challenge of ambiguous music-to-dance mapping and non-natural motion in existing methods.
• To train a model effectively on noisy, weakly supervised skeleton sequences without human-annotated ground truth poses.
• To develop a novel cross-modal evaluation metric that measures similarity between music and dance motion.
• To demonstrate that synthesized dance videos can be indistinguishable from real choreography in user studies.

Proposed method

• The framework uses a two-stage pipeline: first generating a human skeleton sequence from music, then mapping it to a photo-realistic video using a learned pose-to-appearance mapping.
• A music encoder processes 0.1-second audio clips into embeddings, which are fed into a bidirectional GRU and a pose generator to produce skeleton sequences.
• A novel pose perceptual loss is introduced to improve motion plausibility by comparing intermediate feature representations of predicted and reference skeletons.
• Two discriminators are used: a Local Temporal Discriminator to enforce short-term motion coherence and rhythm alignment, and a Global Content Discriminator to ensure long-term motion harmony and emotional consistency.
• A cross-modal evaluation metric is proposed using a pre-trained music feature extractor and a triplet-based metric learning approach to measure similarity between music and generated dance sequences.
• The model is trained end-to-end using adversarial loss, with the pose perceptual loss enabling effective training on noisy OpenPose data without human-verified labels.

Experimental results

Research questions

• RQ1 Can a deep learning model generate realistic, beat-aligned dance videos from arbitrary music without requiring high-quality, human-annotated skeleton data?
• RQ2 How can a pose perceptual loss improve motion plausibility when training on noisy, imperfect skeleton sequences from OpenPose?
• RQ3 To what extent can a dual-discriminator architecture enhance the coherence and rhythm alignment of generated dance sequences?
• RQ4 Can a novel cross-modal evaluation metric effectively measure the semantic similarity between music and dance motion?
• RQ5 How does the synthesized dance video compare to real choreography in terms of perceptual quality and realism?

Key findings

• In a user study with 27 participants, including seven professional dancers, 43.0% of comparisons favored the synthesized dance sequences over ground truth skeletons, indicating perceptual realism comparable to human choreography.
• The proposed pose perceptual loss significantly improves motion plausibility, reducing unnatural poses even when trained on noisy OpenPose data.
• The dual-discriminator setup (local and global) leads to more coherent and rhythmically aligned skeleton sequences than baseline models using only L1/L2 or standard GAN losses.
• The cross-modal evaluation metric successfully estimates similarity between music and dance, with results validated through clustering and embedding analysis using K-means (K=5) and Maximum Mean Discrepancy.
• Quantitative evaluation using BRISQUE shows improved video quality when both the global content discriminator and pose perceptual loss are applied, with lower scores indicating higher perceptual quality.
• Random baselines (Rand Frame and Rand Seq) yielded significantly worse BRISQUE scores, confirming that the model's performance is not due to random frame or sequence selection.