[Paper Review] One-Shot Face Reenactment on Megapixels

Won Jun Kang, Geonsu Lee+2 more|arXiv (Cornell University)|May 26, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes MegaFR, a one-shot, high-resolution face reenactment method that leverages StyleGAN with 3DMM-based rendering images as inputs to enable explicit control over identity, expression, and head pose. By designing a video-free loss function and applying iterative refinement, MegaFR achieves photorealistic megapixel-scale results without requiring high-quality video datasets, outperforming prior methods in disentanglement and visual fidelity.

ABSTRACT

The goal of face reenactment is to transfer a target expression and head pose to a source face while preserving the source identity. With the popularity of face-related applications, there has been much research on this topic. However, the results of existing methods are still limited to low-resolution and lack photorealism. In this work, we present a one-shot and high-resolution face reenactment method called MegaFR. To be precise, we leverage StyleGAN by using 3DMM-based rendering images and overcome the lack of high-quality video datasets by designing a loss function that works without high-quality videos. Also, we apply iterative refinement to deal with extreme poses and/or expressions. Since the proposed method controls source images through 3DMM parameters, we can explicitly manipulate source images. We apply MegaFR to various applications such as face frontalization, eye in-painting, and talking head generation. Experimental results show that our method successfully disentangles identity from expression and head pose, and outperforms conventional methods.

Motivation & Objective

• To address the lack of high-quality, high-resolution video datasets for one-shot face reenactment.
• To overcome the entanglement of identity, expression, and pose in latent space for controllable face manipulation.
• To enable high-resolution (megapixel) face reenactment using only a single source image.
• To develop a method that maintains identity fidelity while handling extreme poses and expressions.
• To demonstrate controllability through applications like face frontalization, eye in-painting, and talking head generation.

Proposed method

• Uses 3DMM-based rendering images—derived from 3DMM parameters—as input to the StyleGAN encoder, providing more interpretable and CNN-friendly features than raw 3DMM parameters.
• Employs a novel loss function that enables training without high-quality video datasets by leveraging perceptual and identity-preserving constraints.
• Applies iterative refinement via ReStyle-style optimization to improve reconstruction quality under extreme poses and expressions.
• Utilizes GAN inversion with pSp, e4e, or PTI to map real source images into StyleGAN latent space while preserving identity details.
• Explicitly controls source images through 3DMM parameters, enabling disentangled manipulation of identity, expression, and pose.
• Integrates StyleGAN with 3DMM priors to enable controllable, high-fidelity face generation and editing.

Experimental results

Research questions

• RQ1 Can face reenactment be achieved at megapixel resolution using only one source image, without relying on large video datasets?
• RQ2 Can 3DMM-based rendering images improve the interpretability and performance of latent space manipulation in StyleGAN for face reenactment?
• RQ3 How effective is a video-free loss function in maintaining identity and photorealism in high-resolution face reenactment?
• RQ4 Can iterative refinement effectively handle extreme facial expressions and head poses in one-shot reenactment?
• RQ5 To what extent can the method be generalized to other face manipulation tasks like frontalization and eye in-painting?

Key findings

• MegaFR achieves state-of-the-art performance in face frontalization, with a Fréchet Inception Distance (FID) of 24.3 when using pSp and 27.2 with e4e, significantly outperforming R&R (88.0) and pSp (62.8).
• The method achieves a perceptual similarity (LPIPS) of 0.23 with pSp and 0.27 with e4e, indicating high perceptual quality in frontalization results.
• Eye in-painting is successfully performed without reference images by directly manipulating 3DMM parameters, enabling control over eye-opening intensity.
• Talking head sequences are generated at high resolution (1024×1024) with accurate lip movement and facial expression transfer, including eye blinking.
• The use of PTI optimization reduces identity drift and preserves fine details in out-of-domain images, as shown in qualitative results.
• MegaFR successfully disentangles identity from expression and pose, enabling explicit control over facial attributes through 3DMM parameters.