[Paper Review] Talking-head Generation with Rhythmic Head Motion

Lele Chen, Guofeng Cui+5 more|arXiv (Cornell University)|Jul 16, 2020

Face recognition and analysis42 references4 citations

TL;DR

This paper proposes a 3D-aware generative network with a hybrid embedding and non-linear composition module to generate photo-realistic, lip-synced talking-head videos with controllable, rhythmic head motion. By modeling head motion and facial expressions explicitly using a short reference video and audio, the method achieves superior temporal coherence and identity preservation over state-of-the-art approaches.

ABSTRACT

When people deliver a speech, they naturally move heads, and this rhythmic head motion conveys prosodic information. However, generating a lip-synced video while moving head naturally is challenging. While remarkably successful, existing works either generate still talkingface videos or rely on landmark/video frames as sparse/dense mapping guidance to generate head movements, which leads to unrealistic or uncontrollable video synthesis. To overcome the limitations, we propose a 3D-aware generative network along with a hybrid embedding module and a non-linear composition module. Through modeling the head motion and facial expressions1 explicitly, manipulating 3D animation carefully, and embedding reference images dynamically, our approach achieves controllable, photo-realistic, and temporally coherent talking-head videos with natural head movements. Thoughtful experiments on several standard benchmarks demonstrate that our method achieves significantly better results than the state-of-the-art methods in both quantitative and qualitative comparisons. The code is available on https://github.com/ lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.

Motivation & Objective

• To generate photo-realistic, lip-synced talking-head videos with natural, rhythmic head motion that conveys prosodic information.
• To overcome limitations of existing methods that produce still faces or rely on sparse/dense supervision for head motion.
• To enable controllable generation where facial expressions are driven by audio and head movements are learned from reference video.
• To improve visual coherence and identity preservation through 3D-aware modeling and dynamic appearance embedding.
• To achieve high-fidelity synthesis without requiring full video frames as input for editing.

Proposed method

• Introduces a 3D-aware module that provides geometric supervision to stabilize GAN training and improve alignment during head motion.
• Employs a hybrid embedding module that dynamically aggregates appearance features from K reference frames to model identity and motion characteristics.
• Uses a non-linear composition module to fuse facial expressions, head motion, and appearance features with precise spatial alignment.
• Learns rhythmic head motion from a 3-second reference video and audio, enabling controllable and natural head pose transitions.
• Decouples head motion and facial expression modeling to improve disentanglement and control.
• Trains end-to-end using audio and reference frames, avoiding reliance on landmark annotations or full video editing.

Experimental results

Research questions

• RQ1 Can a generative model produce photo-realistic talking-head videos with natural, rhythmic head motion without requiring full video frames as input?
• RQ2 How can head motion and facial expression be explicitly disentangled and controlled during audio-driven video generation?
• RQ3 Can 3D-aware modeling improve identity preservation and reduce visual artifacts in dynamic face generation?
• RQ4 Does dynamic reference frame embedding lead to better generalization and visual coherence than global appearance encoding?
• RQ5 How does the proposed method compare in perceptual quality and lip-sync accuracy to state-of-the-art audio- and landmark-driven approaches?

Key findings

• The proposed method achieves state-of-the-art performance on VoxCeleb2 and LRS3-TED datasets in both quantitative metrics and qualitative evaluation.
• The method significantly outperforms prior work in FID, SSIM, and CSIM scores, with CSIM indicating strong generalization across unseen identities.
• User studies show that participants rate videos with natural head motion as significantly more realistic and better synchronized than still-face alternatives.
• Ablation studies confirm that removing the 3D-aware module or non-linear composition block leads to identity drift and visual artifacts near facial edges.
• Performance improves with larger K (number of reference frames), demonstrating the effectiveness of the dynamic embedding strategy.
• The method generates high-quality, temporally coherent videos even under large facial deformations, outperforming methods that rely on landmark or frame-level supervision.