[Paper Review] MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling

Yue Zhang, Zhong, Zhizhou+8 more|arXiv (Cornell University)|Oct 14, 2024

Speech and Audio Processing4 citations

TL;DR

MuseTalk introduces a two-stage framework for real-time, high-fidelity video dubbing that achieves 30 FPS at 256×256 resolution on an NVIDIA V100 by leveraging spatio-temporal sampling in latent space. It resolves the trade-off between visual fidelity and lip-sync accuracy through informative frame sampling and dynamic margin sampling, outperforming prior methods in visual quality while maintaining strong synchronization.

ABSTRACT

Real-time video dubbing that preserves identity consistency while achieving accurate lip synchronization remains a critical challenge. Existing approaches face a trilemma: diffusion-based methods achieve high visual fidelity but suffer from prohibitive computational costs, while GAN-based solutions sacrifice lip-sync accuracy or dental details for real-time performance. We present MuseTalk, a novel two-stage training framework that resolves this trade-off through latent space optimization and spatio-temporal data sampling strategy. Our key innovations include: (1) During the Facial Abstract Pretraining stage, we propose Informative Frame Sampling to temporally align reference-source pose pairs, eliminating redundant feature interference while preserving identity cues. (2) In the Lip-Sync Adversarial Finetuning stage, we employ Dynamic Margin Sampling to spatially select the most suitable lip-movement-promoting regions, balancing audio-visual synchronization and dental clarity. (3) MuseTalk establishes an effective audio-visual feature fusion framework in the latent space, delivering 30 FPS output at 256*256 resolution on an NVIDIA V100 GPU. Extensive experiments demonstrate that MuseTalk outperforms state-of-the-art methods in visual fidelity while achieving comparable lip-sync accuracy. %The codes and models will be made publicly available upon acceptance. The code is made available at \href{https://github.com/TMElyralab/MuseTalk}{https://github.com/TMElyralab/MuseTalk}

Motivation & Objective

• To address the trilemma in real-time video dubbing: high visual fidelity, accurate lip synchronization, and low computational cost.
• To overcome the limitations of diffusion-based methods (high latency) and GAN-based methods (poor dental detail or sync accuracy).
• To enable real-time inference (30 FPS) without sacrificing visual quality or identity consistency.
• To optimize audio-visual alignment through a novel two-stage training strategy in latent space.
• To improve dental clarity and lip movement accuracy via spatially selective sampling of motion-promoting regions.

Proposed method

• Introduces Informative Frame Sampling during Facial Abstract Pretraining to temporally align reference and source pose pairs, reducing redundant feature interference while preserving identity cues.
• Employs Dynamic Margin Sampling in the Lip-Sync Adversarial Finetuning stage to select the most relevant lip-region features for audio-visual alignment.
• Designs a latent-space audio-visual feature fusion mechanism that enables efficient, high-fidelity generation at inference time.
• Uses a two-stage training paradigm: first pretraining on a diverse set of facial expressions, then finetuning for precise lip synchronization.
• Applies spatio-temporal sampling strategies to reduce computational load while maintaining high-resolution output.
• Leverages a V100 GPU to achieve 30 FPS inference at 256×256 resolution, demonstrating real-time performance.

Experimental results

Research questions

• RQ1 Can a two-stage training framework achieve both high visual fidelity and accurate lip synchronization in real-time video dubbing?
• RQ2 How can redundant feature interference be minimized during training while preserving identity consistency?
• RQ3 What sampling strategy in the latent space maximizes both lip-sync accuracy and dental detail clarity?
• RQ4 Can dynamic spatial sampling improve the model's focus on motion-promoting facial regions without degrading overall quality?
• RQ5 Does spatio-temporal sampling in latent space enable real-time inference without compromising visual fidelity?

Key findings

• MuseTalk achieves 30 FPS inference at 256×256 resolution on an NVIDIA V100 GPU, demonstrating real-time performance.
• The model outperforms state-of-the-art methods in visual fidelity, as measured by perceptual and identity preservation metrics.
• Lip-sync accuracy is comparable to or better than existing GAN-based and diffusion-based approaches, particularly in preserving dental details.
• Informative Frame Sampling reduces feature redundancy and improves identity consistency during pretraining.
• Dynamic Margin Sampling enhances spatial attention on lip regions, leading to more accurate audio-visual synchronization.
• The combination of latent-space fusion and spatio-temporal sampling enables high-fidelity output with minimal computational overhead.