[Paper Review] One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

Ting-Chun Wang, Arun Mallya+1 more|arXiv (Cornell University)|Nov 30, 2020

Advanced Vision and ImagingComputer Science97 references22 citations

TL;DR

This paper proposes a one-shot neural talking-head video synthesis model that generates high-quality talking-head videos from a single source image and a driving video using a novel unsupervised 3D keypoint representation. The method enables free-view synthesis by disentangling identity and motion, achieving 10× bandwidth reduction over H.264 while maintaining visual quality comparable to H.264 at CRF 36, and supports dynamic viewpoint rotation and adaptive keypoint compression for video conferencing.

ABSTRACT

We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.

Motivation & Objective

• Address the limitation of fixed viewpoint in 2D-based one-shot talking-head synthesis by enabling local free-view synthesis without 3D graphics models.
• Develop a compact, unsupervised 3D keypoint representation that decouples identity-specific appearance from motion-related dynamics.
• Enable bandwidth-efficient video conferencing by transmitting only the keypoint representation and residuals, reducing data usage by 10× compared to H.264.
• Support dynamic viewpoint manipulation during synthesis, such as head rotation, to simulate face-to-face video conferencing.
• Design adaptive and binary encoding schemes for keypoint metadata and residuals to further reduce bandwidth under variable network conditions.

Proposed method

• Learn a novel 3D keypoint representation from video sequences in an unsupervised manner, decomposing identity and motion information into separate components.
• Use a deep neural network to synthesize talking-head videos by combining the source image’s appearance with motion cues encoded in the 3D keypoints.
• Apply 3D transformations (rotation, translation, deformation) to the keypoint representation to generate novel views during inference.
• Implement an adaptive keypoint scheme that dynamically reduces the number of keypoints based on video complexity, enabling bandwidth adaptation.
• Use a binary latent code encoding scheme for residuals between ground truth and reconstructed frames, reducing transmission cost via arithmetic coding.
• Integrate the residual encoding only when needed (e.g., when source image quality degrades), improving reconstruction fidelity without constant retransmission.

Experimental results

Research questions

• RQ1 Can a one-shot neural talking-head model achieve free-view synthesis without relying on explicit 3D face models?
• RQ2 Can an unsupervised 3D keypoint representation effectively disentangle identity and motion for high-fidelity video synthesis?
• RQ3 To what extent can the proposed keypoint representation reduce bandwidth in video conferencing while preserving visual quality?
• RQ4 How does the adaptive keypoint and residual encoding scheme improve bandwidth efficiency under variable network conditions?
• RQ5 Can the method achieve visual quality comparable to H.264 at significantly lower bitrates, especially in real-time conferencing scenarios?

Key findings

• The proposed method achieves visual quality comparable to H.264 at CRF 36, with a 10× reduction in bandwidth usage on benchmark datasets.
• The model enables local free-view synthesis by applying 3D transformations to the keypoint representation, allowing head rotation and novel-view rendering.
• The adaptive keypoint scheme reduces per-frame metadata size to 53.03 bytes (0.001618 bits per pixel), enabling bandwidth-efficient transmission.
• Binary encoding of residuals requires only 13.40 KB on average per update, significantly reducing retransmission overhead when source image quality degrades.
• User studies on MTurk show that the method is preferred over H.264 at CRF 36 and H.265 at CRF 37, indicating equivalent or better perceptual quality at lower bitrates.
• The method outperforms state-of-the-art one-shot talking-head synthesis models in video reconstruction, motion transfer, and face redirection tasks on benchmark datasets.