[Paper Review] Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator

Chao Xu, Shaoting Zhu+5 more|arXiv (Cornell University)|May 4, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes a unified, diffusion-based generator for multimodal-driven talking face generation that treats the task as target-oriented texture transfer, leveraging CLIP-embedded text for emotion control and a Texture-Geometry-aware Diffusion Model (TGDM) to preserve identity and geometry. The method achieves state-of-the-art performance in both talking face generation and face swapping with stable, GAN-free training and inference.

ABSTRACT

Multimodal-driven talking face generation refers to animating a portrait with the given pose, expression, and gaze transferred from the driving image and video, or estimated from the text and audio. However, existing methods ignore the potential of text modal, and their generators mainly follow the source-oriented feature rearrange paradigm coupled with unstable GAN frameworks. In this work, we first represent the emotion in the text prompt, which could inherit rich semantics from the CLIP, allowing flexible and generalized emotion control. We further reorganize these tasks as the target-oriented texture transfer and adopt the Diffusion Models. More specifically, given a textured face as the source and the rendered face projected from the desired 3DMM coefficients as the target, our proposed Texture-Geometry-aware Diffusion Model decomposes the complex transfer problem into multi-conditional denoising process, where a Texture Attention-based module accurately models the correspondences between appearance and geometry cues contained in source and target conditions, and incorporate extra implicit information for high-fidelity talking face generation. Additionally, TGDM can be gracefully tailored for face swapping. We derive a novel paradigm free of unstable seesaw-style optimization, resulting in simple, stable, and effective training and inference schemes. Extensive experiments demonstrate the superiority of our method.

Motivation & Objective

• Address the instability and poor generalization of GAN-based generators in multimodal talking face generation.
• Overcome the limitations of source-oriented feature rearrangement by redefining the task as target-oriented texture transfer.
• Integrate text modality effectively by leveraging CLIP embeddings for flexible, zero-shot emotion control.
• Unify image, video, audio, and text-driven talking face generation under a single, stable diffusion-based generator.
• Extend the framework to face swapping with a reconstruction-based paradigm that avoids seesaw-style optimization.

Proposed method

• Represent text prompts as CLIP-embedded emotion embeddings to inherit rich semantic semantics for zero-shot emotion control.
• Frame talking face generation as a target-oriented texture transfer: transfer source face texture to a target face rendered from 3DMM coefficients.
• Design a Texture-Geometry-aware Diffusion Model (TGDM) that conditions on both source texture and target geometry via multi-conditional denoising.
• Incorporate a Texture Attention module to model cross-attention between source appearance and target geometry for precise texture transfer.
• Use 3DMM coefficients to generate a 3D-rendered target face as the conditioning pivot, ensuring structural fidelity.
• Adapt the same TGDM framework for face swapping by masking source identity regions and using the rendered target face as a condition, enabling stable reconstruction-based training.

Experimental results

Research questions

• RQ1 Can text modality be effectively leveraged to control emotion in talking face generation without requiring fine-tuning?
• RQ2 Can a unified diffusion-based generator replace task-specific GAN-based architectures for multimodal talking face generation?
• RQ3 Does a target-oriented texture transfer paradigm outperform source-oriented feature rearrangement in preserving identity and reducing artifacts?
• RQ4 Can the proposed TGDM framework be extended to face swapping with stable, GAN-free training and inference?
• RQ5 How does the use of CLIP-embedded text improve zero-shot emotion generalization in facial animation?

Key findings

• The proposed method achieves SOTA performance on both talking face generation and face swapping, with a 15.87 FID score on the FaceForensics++ dataset, outperforming prior methods.
• The method achieves 0.6121 ID-A (identity accuracy) and 0.1122 angle error, significantly outperforming competitors in identity preservation and attribute consistency.
• Abllation studies confirm that using a small mask (excluding only the mouth area) yields better realism than full or dilated masks, reducing artifacts.
• Removing the rendered 3D face as a condition leads to color bias toward the source face, demonstrating its critical role in attribute preservation.
• The TGDM framework enables stable training with a simple reconstruction loss, avoiding the unstable seesaw optimization common in GAN-based face swapping.
• The method generalizes well to unseen emotions via CLIP-guided text prompting, enabling zero-shot emotion control without retraining.