[Paper Review] StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN

Fei Yin, Yong Zhang+8 more|arXiv (Cornell University)|Mar 8, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes StyleHEAT, a unified framework for one-shot high-resolution (1024×1024) editable talking face generation using a pre-trained StyleGAN. It leverages learned latent space properties, motion generators for video/audio driving, a calibration network to correct warping artifacts, and domain loss for feature refinement, enabling disentangled control and flexible editing without requiring high-resolution training data.

ABSTRACT

One-shot talking face generation aims at synthesizing a high-quality talking face video from an arbitrary portrait image, driven by a video or an audio segment. One challenging quality factor is the resolution of the output video: higher resolution conveys more details. In this work, we investigate the latent feature space of a pre-trained StyleGAN and discover some excellent spatial transformation properties. Upon the observation, we explore the possibility of using a pre-trained StyleGAN to break through the resolution limit of training datasets. We propose a novel unified framework based on a pre-trained StyleGAN that enables a set of powerful functionalities, i.e., high-resolution video generation, disentangled control by driving video or audio, and flexible face editing. Our framework elevates the resolution of the synthesized talking face to 1024*1024 for the first time, even though the training dataset has a lower resolution. We design a video-based motion generation module and an audio-based one, which can be plugged into the framework either individually or jointly to drive the video generation. The predicted motion is used to transform the latent features of StyleGAN for visual animation. To compensate for the transformation distortion, we propose a calibration network as well as a domain loss to refine the features. Moreover, our framework allows two types of facial editing, i.e., global editing via GAN inversion and intuitive editing based on 3D morphable models. Comprehensive experiments show superior video quality, flexible controllability, and editability over state-of-the-art methods.

Motivation & Objective

• Address the challenge of generating high-resolution (1024×1024) talking face videos from a single portrait image, despite training data being limited to lower resolutions.
• Overcome the resolution bottleneck in one-shot talking face generation by exploiting the latent space of a pre-trained StyleGAN instead of training from scratch.
• Enable disentangled control over motion via video or audio driving signals, allowing flexible and controllable animation.
• Support both global facial attribute editing via GAN inversion and intuitive editing using 3D morphable models (3DMM) during video generation.
• Minimize visual artifacts from feature warping during motion transfer through a dedicated calibration network and domain loss.

Proposed method

• Utilize the latent feature space of a pre-trained StyleGAN to generate high-resolution faces by applying spatial transformations to latent codes, bypassing the need for high-resolution training data.
• Design a video-based motion generation module that predicts 3DMM parameters from driving video frames and maps them to latent space transformations.
• Implement an audio-based motion generation module that maps audio features to 3DMM parameters for lip-syncing and expression control.
• Introduce a calibration network that refines warped latent features to correct distortion artifacts, especially around eyes and mouth.
• Apply a domain loss to constrain the calibrated features within the original StyleGAN distribution, preserving identity and texture fidelity.
• Support two editing paradigms: GAN inversion for global attribute editing (e.g., age, makeup) and 3DMM-based manipulation for intuitive pose/expression control.

Experimental results

Research questions

• RQ1 Can the latent space of a pre-trained StyleGAN be leveraged to generate 1024×1024 talking face videos without requiring high-resolution training data?
• RQ2 How can motion from video or audio be disentangled and applied to latent space for consistent facial animation?
• RQ3 What mechanisms are effective in correcting warping-induced artifacts during latent space transformation in high-resolution generation?
• RQ4 To what extent can facial attributes be edited globally via GAN inversion while preserving motion quality and identity?
• RQ5 Can intuitive editing via 3DMM parameters be effectively integrated into a high-resolution, one-shot talking face framework?

Key findings

• StyleHEAT achieves the first 1024×1024 resolution in one-shot talking face generation, significantly exceeding prior state-of-the-art methods limited to 512×512 or lower.
• The calibration network reduces visible artifacts around the eyes and mouth, enabling correct generation of closed eyes and teeth, which are commonly distorted in baseline warping approaches.
• The domain loss improves visual quality by preventing over-regularization, reducing blurriness and preserving fine details like wrinkles and hair texture.
• Global editing via GAN inversion successfully modifies attributes such as beard, makeup, and age without interfering with motion transfer, maintaining temporal consistency.
• Intuitive editing using 3DMM parameters enables consistent pose and expression control, with superior texture and lighting fidelity compared to 3DMM-based baselines like PIRenderer.
• Ablation studies confirm that both the calibration network and domain loss are essential for high-fidelity results, with their removal leading to significant degradation in quality and realism.