[Paper Review] Everybody's Talkin': Let Me Talk as You Want

Linsen Song, Wayne Wu+3 more|arXiv (Cornell University)|Jan 15, 2020

Generative Adversarial Networks and Image Synthesis59 references35 citations

TL;DR

The paper presents an end-to-end framework that edits a target portrait video by translating source audio into target-expression parameters, then renders a photo-realistic video with a neural video rendering network, enabling many-to-many audio-to-video translation without identity-specific networks.

ABSTRACT

We present a method to edit a target portrait footage by taking a sequence of audio as input to synthesize a photo-realistic video. This method is unique because it is highly dynamic. It does not assume a person-specific rendering network yet capable of translating arbitrary source audio into arbitrary video output. Instead of learning a highly heterogeneous and nonlinear mapping from audio to the video directly, we first factorize each target video frame into orthogonal parameter spaces, i.e., expression, geometry, and pose, via monocular 3D face reconstruction. Next, a recurrent network is introduced to translate source audio into expression parameters that are primarily related to the audio content. The audio-translated expression parameters are then used to synthesize a photo-realistic human subject in each video frame, with the movement of the mouth regions precisely mapped to the source audio. The geometry and pose parameters of the target human portrait are retained, therefore preserving the context of the original video footage. Finally, we introduce a novel video rendering network and a dynamic programming method to construct a temporally coherent and photo-realistic video. Extensive experiments demonstrate the superiority of our method over existing approaches. Our method is end-to-end learnable and robust to voice variations in the source audio.

Motivation & Objective

• Motivate and enable autoregressive audio-based editing of portrait videos across arbitrary source and target identities.
• Disentangle video frames into geometry, pose, and expression to facilitate robust audio-to-video translation.
• Develop an Audio ID-Removing Network to makes audio-to-expression translation identity-agnostic.
• Propose a Neural Video Rendering Network for mouth-region completion guided by mouth landmarks to ensure realism and temporal coherence.

Proposed method

• Parse each frame with monocular 3D face reconstruction to obtain geometry, expression, and pose (3DMM-based parameters).
• Use an Audio-to-Expression Translation Network (LSTM-based) to map speaker-agnostic audio features to expression parameters.
• Incorporate an Audio ID-Removing Network to remove speaker identity from audio features before translation.
• Formulate mouth-region generation as a face completion problem conditioned on mouth landmarks and a masked frame.
• Apply a Neural Video Rendering Network with a Unet-like architecture and landmark heatmaps, trained with reconstruction, adversarial, perceptual, and temporal losses.

Experimental results

Research questions

• RQ1 Can audio from arbitrary speakers drive photo-realistic portrait videos of arbitrary target identities without per-person training?
• RQ2 Does decomposing video frames into geometry, pose, and expression improve cross-speaker audio-to-video translation?
• RQ3 Does an identity-agnostic audio representation improve lip synchronization and visual realism across diverse voices?

Key findings

• The proposed 3D parameter-based misalignment approach enables many-to-many audio-to-video translation with a single generator across multiple identities.
• Audio ID-Removing Network improves lip synchronization and reduces identity leakage in audio features.
• Jointly trained completion-based mouth generation with landmark guidance yields competitive PSNR/SSIM and improved realism over baselines.
• The method supports large pose variation and audio editing/singing, demonstrating robust generalization to unseen speakers and languages.
• Compared to state-of-the-art methods, the approach offers better texture details and seamless background blending in many scenarios.