[Paper Review] DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer

Zhiyuan Ma, Xiangyu Zhu+4 more|arXiv (Cornell University)|Feb 8, 2024

Face recognition and analysis4 citations

TL;DR

DiffSpeaker proposes a diffusion-based Transformer model with biased conditional self/cross-attention for speech-driven 3D facial animation, achieving state-of-the-art performance in lip synchronization and facial expression quality while enabling fast, parallel inference. The method addresses data scarcity in short audio-4D sequences by embedding diffusion step and speaking style biases into attention mechanisms, improving both alignment and diversity without requiring large-scale paired data.

ABSTRACT

Speech-driven 3D facial animation is important for many multimedia applications. Recent work has shown promise in using either Diffusion models or Transformer architectures for this task. However, their mere aggregation does not lead to improved performance. We suspect this is due to a shortage of paired audio-4D data, which is crucial for the Transformer to effectively perform as a denoiser within the Diffusion framework. To tackle this issue, we present DiffSpeaker, a Transformer-based network equipped with novel biased conditional attention modules. These modules serve as substitutes for the traditional self/cross-attention in standard Transformers, incorporating thoughtfully designed biases that steer the attention mechanisms to concentrate on both the relevant task-specific and diffusion-related conditions. We also explore the trade-off between accurate lip synchronization and non-verbal facial expressions within the Diffusion paradigm. Experiments show our model not only achieves state-of-the-art performance on existing benchmarks, but also fast inference speed owing to its ability to generate facial motions in parallel.

Motivation & Objective

• To improve speech-driven 3D facial animation by combining diffusion models and Transformers, overcoming limitations of prior methods.
• To address the challenge of limited paired audio-4D data in training diffusion-based Transformers for facial animation.
• To balance accurate lip synchronization with expressive, non-verbal facial movements within a conditional diffusion framework.
• To enable fast, parallel inference by designing attention mechanisms that reduce data intensity while preserving long-range context.

Proposed method

• Introduces a novel biased conditional self-attention and cross-attention mechanism that embeds diffusion step and speaking style as conditioning biases to guide attention toward relevant temporal and semantic cues.
• Uses a conditional diffusion model where the denoising process is guided by audio input, with a 10% chance of unconditional denoising during training to improve output diversity.
• Applies a guidance scale $w$ to control the trade-off between audio-visual alignment (lip sync) and facial expression richness, though the final model disables this for stable, high-accuracy synchronization.
• Designs task-specific biases in attention to prioritize temporal alignment between speech and facial motion, reducing the need for extensive data during training.
• Replaces standard self/cross-attention with condition-aware biases that steer attention toward relevant audio and motion tokens, improving efficiency and performance on short, limited data.
• Employs a denoising U-Net backbone with cross-attention between audio and motion tokens, enhanced by fixed biases that encode diffusion step and speaking style information.

Experimental results

Research questions

• RQ1 How can a diffusion-based Transformer be effectively trained on limited paired audio-4D facial motion data for speech-driven animation?
• RQ2 What role do condition-specific biases in self/cross-attention play in improving temporal alignment and expression quality in diffusion-based facial animation?
• RQ3 To what extent does unconditional guidance improve facial expression diversity at the cost of lip synchronization accuracy?
• RQ4 Can a diffusion-based Transformer achieve fast, parallel inference while maintaining high-quality facial animation?
• RQ5 How do task-specific attention biases compare to generic or uniform biasing schemes in terms of performance and robustness?

Key findings

• DiffSpeaker achieves state-of-the-art performance on existing benchmarks, outperforming prior methods in both Facial Detail Distance (FDD) and Lip Video Error (LVE) metrics.
• The model achieves fast inference speed by generating entire facial motion sequences in parallel, with average latency for 10–90s audio clips significantly lower than autoregressive baselines.
• Removing cross-attention bias leads to a substantial drop in performance, increasing LVE by 12.3% and FDD by 8.7%, indicating its critical role in audio-motion alignment.
• The inclusion of self-attention bias improves modeling of temporal continuity, reducing motion jitter and enhancing facial dynamics realism.
• Using FaceFormer-style biases instead of task-specific ones results in suboptimal performance, confirming the necessity of condition-aware bias design.
• The model maintains stable results across random seeds, with narrow confidence intervals, indicating robustness and reliability without relying on unconditional guidance.