[Paper Review] Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Ran Yi, Zipeng Ye+3 more|arXiv (Cornell University)|Feb 24, 2020

Face recognition and analysis60 references77 citations

TL;DR

This paper presents a neural pipeline that maps audio from a source to a talking-face video of a target with personalized head pose, using 3D face reconstruction and a memory-augmented GAN for realism. A general audio-to-face mapping is fine-tuned on a short target video to achieve personalized motion and head pose.

ABSTRACT

Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

Motivation & Objective

• Motivate natural talking-face video generation with personalized head pose instead of fixed pose.
• Learn a general audio-to-face mapping and fine-tune it on a short target video to capture individualized head motion and expressions.
• Bridge audio–visual cues through 3D facial animation and rendering to produce realistic frames.
• Refine rendered frames with a memory-augmented GAN that adapts to arbitrary target identities.

Proposed method

• Stage 1: Learn a general audio-to-expression-and-head-pose mapping from audio MFCC features to 3DMM expression and pose using an LSTM network.
• Stage 2: Reconstruct the target’s 3D face from a short video, fine-tune the mapping to capture personalized talking behavior, and obtain stage-2 3D facial animation.
• Render 3D facial animation into frames using target identity texture/illumination, then refine with a memory-augmented GAN that uses identity features and a memory module to adapt across identities.
• Use a memory network to store and retrieve identity features for refinement, enabling one-shot/few-shot personalization.
• Train the GAN with a two-stream conditioning: a window of rendered frames and an identity feature, employing an attention-based generator and PatchGAN-based discriminator.

Experimental results

Research questions

• RQ1 Can audio alone drive natural lip synchronization while incorporating personalized head pose based on a short target video?
• RQ2 How can 3D geometry and rendering be integrated with learning-based refinement to produce realistic talking-face videos for arbitrary identities?
• RQ3 Does a memory-augmented GAN enable high-quality, identity-aware frame refinement across different subjects?
• RQ4 What is the effect of fine-tuning on a small target video (~300 frames) for personalized head pose adaptation?

Key findings

• The proposed Ours-P model achieves better image quality, lip synchronization, and naturalness than state-of-the-art methods in subjective user studies.
• Quantitative results on the LRW dataset show Ours-G achieving the highest PSNR (30.94) and competitive SSIM (0.75) and LMD (1.58) compared to Chen, Wiles, You said that, DAVS, and ATVG.
• The method demonstrates good lip synchronization with a comparable or superior quantitative profile to prior methods when using a single input frame (Ours-G).
• Fine-tuning with roughly 300 frames (Ours-P) enables personalized head pose and expressions that outperform fixed-pose baselines in qualitative and user-study assessments.
• A memory-augmented GAN with identity-conditioned refinement yields more realistic textures and facial details across diverse identities than non-memory baselines.