[Paper Review] DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video

Zhimeng Zhang, Zhipeng Hu+4 more|arXiv (Cornell University)|Mar 7, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes DINet, a Deformation Inpainting Network for high-fidelity, photo-realistic face visually dubbing on high-resolution videos. By spatially deforming reference facial feature maps to align with audio-driven mouth shapes and head poses, then inpainting the mouth region using a learned decoder, DINet preserves high-frequency textural details better than direct generation methods, achieving state-of-the-art results in both qualitative and quantitative evaluations on high-resolution benchmarks.

ABSTRACT

For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists of one deformation part and one inpainting part. In the first part, five reference facial images adaptively perform spatial deformation to create deformed feature maps encoding mouth shapes at each frame, in order to align with the input driving audio and also the head poses of the input source images. In the second part, to produce face visually dubbing, a feature decoder is responsible for adaptively incorporating mouth movements from the deformed feature maps and other attributes (i.e., head pose and upper facial expression) from the source feature maps together. Finally, DINet achieves face visually dubbing with rich textural details. We conduct qualitative and quantitative comparisons to validate our DINet on high-resolution videos. The experimental results show that our method outperforms state-of-the-art works.

Motivation & Objective

• To address the challenge of generating photo-realistic face visually dubbing on high-resolution videos with few-shot learning.
• To overcome the limitations of direct pixel generation methods that fail to preserve high-frequency textural details in mouth regions.
• To improve lip synchronization and facial realism by leveraging spatial deformation of reference image features instead of end-to-end generation.
• To maintain identity, head pose, and upper facial expression consistency while accurately aligning mouth movements with driving audio.
• To achieve superior performance on high-resolution video datasets compared to existing state-of-the-art methods.

Proposed method

• DINet consists of two main components: a deformation part and an inpainting part.
• The deformation part uses adaptive spatial transformation (AdaAT) to deform feature maps from five reference facial images based on driving audio and source face head pose.
• AdaAT performs region-level deformation with regularization, preserving structural and textural details better than pixel-level methods like attention or dense flow.
• The inpainting part fuses deformed mouth features with source face features (including head pose and upper facial expression) via a convolutional decoder to reconstruct the mouth region.
• The network is trained with perceptual, adversarial, and audio-visual synchronization losses to ensure realism and alignment.
• Ablation studies validate the effectiveness of deformation over direct generation and the superiority of AdaAT over attention and dense flow for feature deformation.

Experimental results

Research questions

• RQ1 Can spatial deformation of reference image features improve high-frequency texture preservation in high-resolution face visually dubbing?
• RQ2 How does deformation-based feature manipulation compare to direct pixel generation in terms of visual fidelity and lip synchronization?
• RQ3 What is the impact of different deformation mechanisms (e.g., AdaAT vs. attention vs. dense flow) on texture preservation and artifact generation?
• RQ4 To what extent can a few-shot method maintain identity and head pose consistency while generating realistic mouth movements from audio?
• RQ5 Does the proposed deformation-inpainting framework outperform existing state-of-the-art methods on high-resolution video benchmarks?

Key findings

• DINet achieves the highest SSIM (0.9425), PSNR (30.0082), and lowest LPIPS (0.0289) on the HDTF dataset, outperforming all ablation variants and SOTA methods.
• The ablation study shows that removing the deformation component reduces SSIM to 0.9147 and PSNR to 26.1665, indicating its critical role in preserving details.
• Omitting the inpainting module results in visible artifacts and reduced quality (SSIM: 0.8691, PSNR: 26.0071), confirming the necessity of feature fusion.
• Using attention or dense flow for deformation leads to higher LPIPS (0.0433 and 0.0656) and more artifacts, especially in facial structures like eyes and sight lines.
• User study shows DINet achieves the highest realism score (3.4 out of 5), surpassing Wav2Lip (2.4), ATVG (2.9), MakeItTalk (3.1), and PC-AVS (3.3).
• The method effectively preserves fine textures such as beard and nasolabial folds, which are often blurred in direct generation approaches.