[Paper Review] Third Time's the Charm? Image and Video Editing with StyleGAN3

Yuval Alaluf, Or Patashnik+5 more|arXiv (Cornell University)|Jan 31, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper investigates StyleGAN3 for image and video editing, demonstrating that aligned training data enhances disentangled editing and inversion despite StyleGAN3's capability to generate unaligned images from unaligned data. The authors propose a novel video inversion and editing pipeline leveraging StyleGAN3's translation/rotation equivariance to reduce texture sticking and expand field of view, achieving high-fidelity edits on real images and videos using encoder-based inversion and fine-tuned generators.

ABSTRACT

StyleGAN is arguably one of the most intriguing and well-studied generative models, demonstrating impressive performance in image generation, inversion, and manipulation. In this work, we explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks. In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery. Next, our analysis of the disentanglement of the different latent spaces of StyleGAN3 indicates that the commonly used W/W+ spaces are more entangled than their StyleGAN2 counterparts, underscoring the benefits of using the StyleSpace for fine-grained editing. Considering image inversion, we observe that existing encoder-based techniques struggle when trained on unaligned data. We therefore propose an encoding scheme trained solely on aligned data, yet can still invert unaligned images. Finally, we introduce a novel video inversion and editing workflow that leverages the capabilities of a fine-tuned StyleGAN3 generator to reduce texture sticking and expand the field of view of the edited video.

Motivation & Objective

• To analyze the disentanglement properties of StyleGAN3's latent spaces compared to StyleGAN2.
• To evaluate the performance of existing encoder-based inversion techniques on StyleGAN3, especially with unaligned data.
• To develop a new video editing workflow that leverages StyleGAN3's equivariance to improve temporal consistency and reduce artifacts.
• To investigate whether aligned or unaligned training data yields better results for editing and inversion in StyleGAN3.
• To enable high-fidelity editing of real images and videos using a fine-tuned StyleGAN3 generator and a dedicated encoding scheme.

Proposed method

• Train a StyleGAN3 generator on aligned data to improve disentanglement and inversion performance, even though it can generate unaligned images from unaligned data.
• Propose a new encoder-based inversion method trained solely on aligned data but capable of inverting unaligned real images.
• Leverage StyleGAN3’s translation and rotation equivariance to design a video inversion and editing pipeline that reduces texture sticking.
• Introduce a field-of-view expansion technique by manipulating latent codes to reconstruct cropped or partially visible regions in video frames.
• Fine-tune StyleGAN3 using StyleGAN-NADA for style transfer and semantic editing, preserving latent space alignment across parent and child generators.
• Apply off-the-shelf editing techniques (e.g., InterFaceGAN, StyleCLIP) to latent codes obtained via the proposed inversion pipeline for real image and video editing.

Experimental results

Research questions

• RQ1 How does the disentanglement of StyleGAN3’s latent spaces compare to that of StyleGAN2, particularly in the W and W+ spaces?
• RQ2 Can existing encoder-based inversion techniques effectively invert unaligned real images when applied to StyleGAN3?
• RQ3 Does training StyleGAN3 on aligned data improve editing and inversion performance compared to unaligned training, despite its ability to generate unaligned images?
• RQ4 Can StyleGAN3’s equivariance properties be exploited to reduce texture sticking and improve temporal consistency in video editing?
• RQ5 Can a video editing pipeline be designed that combines inversion, editing, and field-of-view expansion using StyleGAN3’s architectural strengths?

Key findings

• The W and W+ latent spaces in StyleGAN3 are more entangled than in StyleGAN2, indicating that the StyleSpace is better suited for fine-grained editing.
• Existing encoder-based inversion methods struggle with unaligned data when applied to StyleGAN3, but a new encoder trained only on aligned data achieves comparable reconstruction quality on unaligned images.
• StyleGAN3 trained on aligned data enables superior disentangled editing and inversion performance, even though unaligned training also supports unaligned generation.
• The proposed video editing pipeline reduces texture sticking by leveraging StyleGAN3’s translation and rotation equivariance, resulting in more temporally consistent edits.
• The field-of-view expansion technique successfully reconstructs cut-off facial regions (e.g., chin, hair) in cropped video frames by manipulating latent codes.
• Fine-tuned StyleGAN3 generators trained with StyleGAN-NADA preserve semantic alignment in the latent space, enabling consistent editing across parent and child models.