[Paper Review] Audio-driven Talking Face Video Generation with Natural Head Pose

Ran Yi, Zipeng Ye+3 more|arXiv (Cornell University)|Feb 24, 2020

Face recognition and analysis16 citations

TL;DR

This paper proposes a deep neural network that generates high-quality talking face videos with personalized, natural head poses by leveraging audio and a short target video. It reconstructs 3D face animation and uses a memory-augmented GAN to refine frames, achieving realistic head movements and lip sync with only ~300 frames of fine-tuning data.

ABSTRACT

Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

Motivation & Objective

• To address the limitation of existing talking face video methods that fix head pose, by enabling natural, personalized head movements.
• To model in-plane and out-of-plane head rotations caused by natural speech, which are challenging for realistic synthesis.
• To generate high-fidelity talking face videos with synchronized lip movements and expressive facial dynamics using only a short target video.
• To develop a fine-tuning strategy that learns personalized talking behavior—including head pose—efficiently from a small number of frames (~300).

Proposed method

• The model takes audio A and a short video V of a target person as inputs to generate a personalized talking face video with dynamic head pose.
• It reconstructs 3D face animation from the audio and visual input, enabling realistic in-plane and out-of-plane head rotations.
• A memory-augmented GAN module is introduced to refine synthesized frames, ensuring smooth background transitions and photorealistic quality.
• A two-stage training strategy is used: first pre-training on a public dataset, then fine-tuning on the target person’s short video to adapt to individual head movement patterns.
• The method leverages visual cues from the short video V to infer personalized head pose behavior, even with limited frames.
• Lip synchronization is achieved by jointly modeling audio and visual signals, ensuring temporal alignment between speech and mouth movements.

Experimental results

Research questions

• RQ1 Can a deep learning model generate talking face videos with natural, personalized head movements using only a short video of the target person?
• RQ2 How effectively can 3D face animation reconstruction and GAN-based refinement improve realism in synthesized talking face videos with dynamic head poses?
• RQ3 To what extent does fine-tuning on a small number of frames (~300) enable accurate modeling of individual head movement patterns?
• RQ4 How does the proposed method compare to state-of-the-art approaches in terms of lip synchronization, facial expression, and head movement naturalness?

Key findings

• The proposed method generates talking face videos with significantly more distinguishing and natural head movement effects compared to state-of-the-art methods.
• User studies confirm that the generated videos are perceived as more realistic and visually compelling due to accurate head pose dynamics.
• The method achieves high-quality lip synchronization by effectively aligning audio and visual modalities through joint modeling.
• Fine-tuning using only ~300 frames of the target person’s video is sufficient to learn personalized talking behavior, including complex head pose patterns.
• The memory-augmented GAN module successfully improves frame realism and ensures smooth background transitions in the synthesized video.
• Extensive experiments demonstrate consistent superiority in visual quality and motion naturalness across diverse test cases.