[Paper Review] DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model

Fan Zhang, Naye Ji+2 more|arXiv (Cornell University)|Jan 24, 2023

Hand Gesture Recognition Systems4 citations

TL;DR

This paper proposes DiffMotion, a diffusion-based generative model for speech-driven 3D gesture synthesis that uses an autoregressive temporal encoder and a denoising diffusion module to learn complex, one-to-many mappings between speech and gestures. The method generates diverse, natural, and rhythmically synchronized gestures, outperforming state-of-the-art baselines in both objective and subjective evaluations, with human ratings nearly matching real motion-capture data.

ABSTRACT

Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the temporal context of the speech input and historical gestures. The diffusion module learns a parameterized Markov chain to gradually convert a simple distribution into a complex distribution and generates the gestures according to the accompanied speech. Compared with baselines, objective and subjective evaluations confirm that our approach can produce natural and diverse gesticulation and demonstrate the benefits of diffusion-based models on speech-driven gesture synthesis.

Motivation & Objective

• To address the inherent one-to-many mapping challenge between speech and gestures in virtual human animation.
• To improve the naturalness and diversity of speech-driven gesture synthesis beyond deterministic or autoencoder-based models.
• To explore the potential of diffusion models in cross-modal gesture generation, where they have seen limited prior use.
• To develop a fully end-to-end, weakly supervised framework requiring no manual gesture annotation.
• To generate high-quality, temporally coherent, and expressive gestures that match speech rhythm and semantics.

Proposed method

• An autoregressive temporal encoder processes sequential speech and historical gesture features to capture long-range temporal dependencies.
• A denoising diffusion probabilistic model gradually denoises a latent representation from Gaussian noise to generate realistic 3D gesture sequences.
• The diffusion module is conditioned on speech embeddings and prior gesture states, enabling conditional generation of diverse outputs.
• The model is trained via variational lower bound optimization to minimize reconstruction error and improve sample quality.
• The architecture enables end-to-end training on unstructured, raw motion-capture data without manual annotation.
• Inference involves reverse denoising steps, with hyperparameter tuning of diffusion step count to balance quality and speed.

Figure 1: Random samples from DiffMotion can give many distinct yet natural output gestures within and between sequences, even if the input speech audio is the same.

Experimental results

Research questions

• RQ1 Can a diffusion-based model effectively learn the complex, one-to-many mapping between speech and gestures?
• RQ2 How does the proposed DiffMotion model compare to existing generative models (e.g., GANs, VAEs, normalizing flows) in generating diverse and natural gestures?
• RQ3 What is the impact of the number of diffusion steps on gesture quality, diversity, and inference speed?
• RQ4 Can the model generate gestures that are both realistic and rhythmically aligned with speech, even when not present in the training data?
• RQ5 To what extent does the model’s performance match or approach real motion-capture data in human evaluation?

Key findings

• DiffMotion achieved human-likeness scores of 3.89 ± 0.95, nearly matching real motion-capture data (3.89 ± 0.93), indicating high perceptual realism.
• The model scored 3.93 ± 0.93 on appropriateness, showing strong alignment between gestures and speech content.
• On diversity evaluation, DiffMotion (3.91 ± 0.96) significantly outperformed ground truth (3.60 ± 1.02), generating more varied gestures than present in the dataset.
• The optimal number of diffusion steps was found to be 100, balancing motion continuity and inference efficiency.
• Inference time increased significantly beyond N=1000, with occasional generation of bizarre poses due to over-smoothing of fine details.
• The model achieved a validation loss of 0.10 at N=2500, but with a 8.02-second per-frame inference time, highlighting a trade-off between quality and speed.

Figure 2: DiffMotion schematic. The model consists of an Autoregressive Temporal Encoder and a Denoising Diffusion Probabilistic Module.