[Paper Review] Generate and Edit Your Own Character in a Canonical View

Jeong-gi Kwak, Yuanming Li+3 more|arXiv (Cornell University)|May 6, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a unified, 3D-supervision-free framework that generates and edits stylized portraits in a canonical frontal view using latent space manipulation in StyleGAN. By training a novel latent mapper on pseudo-ground-truth pairs from a NeRF-based GAN, the method achieves simultaneous stylization and frontalization, outperforming cascaded approaches in visual quality, identity preservation, and inference speed with a 65.21 FID and 0.26s runtime.

ABSTRACT

Recently, synthesizing personalized characters from a single user-given portrait has received remarkable attention as a drastic popularization of social media and the metaverse. The input image is not always in frontal view, thus it is important to acquire or predict canonical view for 3D modeling or other applications. Although the progress of generative models enables the stylization of a portrait, obtaining the stylized image in canonical view is still a challenging task. There have been several studies on face frontalization but their performance significantly decreases when input is not in the real image domain, e.g., cartoon or painting. Stylizing after frontalization also results in degenerated output. In this paper, we propose a novel and unified framework which generates stylized portraits in canonical view. With a proposed latent mapper, we analyze and discover frontalization mapping in a latent space of StyleGAN to stylize and frontalize at once. In addition, our model can be trained with unlabelled 2D image sets, without any 3D supervision. The effectiveness of our method is demonstrated by experimental results.

Motivation & Objective

• To address the challenge of generating stylized portraits in canonical frontal view from arbitrary-angle input images without 3D supervision.
• To overcome the degradation in identity and visual quality when applying frontalization followed by stylization in cascade.
• To discover an accurate latent space mapping for frontalization that preserves the semantic manifold of StyleGAN.
• To enable end-to-end stylization and frontalization using only unlabelled 2D image sets, avoiding reliance on 3DMM or pose estimators.
• To maintain compatibility with existing StyleGAN-based latent manipulation techniques for downstream editing.

Proposed method

• A NeRF-based GAN is trained from scratch to generate pseudo-ground-truth pairs of random-view and frontal-view images, providing supervision for latent mapping.
• A novel latent mapper is introduced to learn an accurate mapping from arbitrary-view latent codes to canonical frontal-view latent codes in StyleGAN’s latent space.
• The latent mapper is trained using a multi-loss objective combining L1, perceptual, and identity-preserving losses to ensure fidelity and realism.
• Stylization is achieved via GAN interpolation, where the manipulated latent code affects only the early layers of the StyleGAN generator to preserve identity and quality.
• The method operates entirely in the latent space, enabling direct application of existing latent editing techniques like InterFaceGAN for attribute manipulation.
• The framework is trained end-to-end on unlabelled 2D image sets, eliminating the need for annotated landmarks, 3D meshes, or pre-trained 3D models.

Experimental results

Research questions

• RQ1 Can a unified framework generate stylized portraits in canonical frontal view without 3D supervision or annotated labels?
• RQ2 How can accurate frontalization be achieved in the latent space of StyleGAN without relying on 3D geometry or pose estimation?
• RQ3 Does joint stylization and frontalization in a single step outperform cascaded pipelines (e.g., frontalize then stylize) in visual quality and identity preservation?
• RQ4 To what extent can the proposed method maintain identity and semantic consistency across diverse stylization domains?
• RQ5 Can the method be seamlessly integrated with existing latent space editing techniques for attribute manipulation in the canonical view?

Key findings

• The proposed method achieves a Fréchet Inception Distance (FID) of 65.21, significantly outperforming cascaded approaches (95.25 and 86.64) in visual quality.
• The method achieves the fastest inference speed at 0.26 seconds per output, outperforming cascaded methods (1.84s and 4.63s) and enabling real-time application.
• The method preserves identity better than cascaded pipelines, achieving a 15.5% higher identity score (82.26) than the next best method (77.10) on the ArcFace metric.
• Qualitative results show superior photorealism and identity consistency, with no visible artifacts or distortion, even on stylized inputs like cartoons and paintings.
• The method successfully enables editing of semantic attributes (e.g., age, gender, smile) in the canonical view using InterFaceGAN, demonstrating compatibility with existing latent editing tools.
• The ablation study confirms that the latent mapper’s design is critical, as removing it leads to significant degradation in identity and visual quality.