[Paper Review] HeadGAN: Video-and-Audio-Driven Talking Head Synthesis

Michail Christos Doukas, Stefanos Zafeiriou+1 more|arXiv (Cornell University)|Dec 15, 2020

Face recognition and analysis4 citations

TL;DR

HeadGAN proposes a GAN-based talking head synthesis method that leverages 3D face representations from driving videos and audio features to improve identity preservation and photo-realism, especially under extreme head poses. The approach achieves more plausible mouth movements and realistic facial reenactment by conditioning the generator on both 3D facial geometry and audio cues.

ABSTRACT

Recent attempts to solve the problem of talking head synthesis using a single reference image have shown promising results. However, most of them fail to meet the identity preservation problem, or perform poorly in terms of photo-realism, especially in extreme head poses. We propose HeadGAN, a novel reenactment approach that conditions synthesis on 3D face representations, which can be extracted from any driving video and adapted to the facial geometry of any source. We improve the plausibility of mouth movements, by utilising audio features as a complementary input to the Generator. Quantitative and qualitative experiments demonstrate the merits of our approach.

Motivation & Objective

• Address the identity preservation problem in single-image talking head synthesis.
• Improve photo-realism, particularly in extreme head poses, where prior methods fail.
• Enhance the realism of mouth movements by incorporating audio features as a conditioning signal.
• Enable robust facial reenactment by adapting 3D face representations to the source identity.

Proposed method

• Condition the generator on 3D face representations extracted from a driving video to preserve identity and geometry.
• Adapt the 3D face representation to match the facial structure of the source image for identity alignment.
• Integrate audio features as an additional input to the generator to guide realistic lip movements.
• Utilize a GAN framework where the discriminator enforces perceptual quality and identity consistency.
• Train the generator to synthesize video frames that align with both visual motion and audio-driven speech.
• Leverage a two-stage training process: first, extract 3D face geometry from the driving video; second, generate frames conditioned on source identity, 3D geometry, and audio.

Experimental results

Research questions

• RQ1 Can 3D face representations from a driving video improve identity preservation in talking head synthesis?
• RQ2 Does incorporating audio features lead to more realistic and synchronized lip movements?
• RQ3 How does the proposed method perform under extreme head poses compared to prior approaches?
• RQ4 To what extent does the combination of 3D geometry and audio enhance photo-realism in generated talking heads?

Key findings

• HeadGAN achieves superior identity preservation by aligning 3D face representations from the driving video with the source image.
• The integration of audio features significantly improves the realism and synchronization of mouth movements.
• The method demonstrates robust performance under extreme head poses, outperforming prior approaches in visual plausibility.
• Quantitative and qualitative evaluations confirm enhanced photo-realism and temporal consistency in generated videos.