[Paper Review] 3D Cartoon Face Generation with Controllable Expressions from a Single GAN Image

Hao Wang, Shen, Wenhao+3 more|arXiv (Cornell University)|Jul 29, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes a novel GAN-based framework to generate 3D cartoon face avatars from a single 2D GAN-generated human face image without 3D supervision. By leveraging disentangled StyleGAN latent space manipulation and optimized latent code offsets, the method enables controllable facial expressions, poses, and lighting, achieving high-fidelity 3D reconstruction with identity preservation and stylized appearance transfer.

ABSTRACT

In this paper, we investigate an open research task of generating 3D cartoon face shapes from single 2D GAN generated human faces and without 3D supervision, where we can also manipulate the facial expressions of the 3D shapes. To this end, we discover the semantic meanings of StyleGAN latent space, such that we are able to produce face images of various expressions, poses, and lighting conditions by controlling the latent codes. Specifically, we first finetune the pretrained StyleGAN face model on the cartoon datasets. By feeding the same latent codes to face and cartoon generation models, we aim to realize the translation from 2D human face images to cartoon styled avatars. We then discover semantic directions of the GAN latent space, in an attempt to change the facial expressions while preserving the original identity. As we do not have any 3D annotations for cartoon faces, we manipulate the latent codes to generate images with different poses and lighting conditions, such that we can reconstruct the 3D cartoon face shapes. We validate the efficacy of our method on three cartoon datasets qualitatively and quantitatively.

Motivation & Objective

• To enable 3D cartoon face generation from a single 2D GAN-generated human face image without requiring 3D annotations or multi-view data.
• To achieve controllable facial expressions in the generated 3D avatars while preserving the original identity and appearance.
• To bridge the semantic consistency between 2D human faces, stylized cartoon images, and reconstructed 3D shapes through shared latent space representation.
• To develop a method that enables expression, pose, and lighting manipulation via optimized latent code offsets in StyleGAN's W space.
• To demonstrate high-quality 3D reconstruction using pseudo-samples from manipulated latent codes under varying viewpoints and lighting.

Proposed method

• Fine-tune a pretrained StyleGAN model on cartoon datasets to learn a shared latent space between human faces and cartoon avatars.
• Use GAN inversion to project real or GAN-generated 2D human faces into the StyleGAN latent space W, enabling identity-preserving stylization.
• Apply closed-form factorization (SeFa) to discover initial latent code offsets for facial expression manipulation.
• Optimize the initial offsets using face identity loss and low-level feature loss to ensure only expression changes while preserving identity and shape.
• Generate pseudo-samples with varied poses and lighting by manipulating latent codes, which are used as supervisory signals for 3D reconstruction.
• Reconstruct 3D shapes using neural rendering from the manipulated 2D images, leveraging normal maps and multi-view consistency.

Experimental results

Research questions

• RQ1 Can 3D cartoon face shapes be reconstructed from a single 2D GAN-generated human face image without any 3D supervision or annotations?
• RQ2 How can facial expressions be manipulated in the generated 3D avatars while preserving the original identity and appearance?
• RQ3 To what extent can disentangled latent space manipulation in StyleGAN enable control over expression, pose, and lighting in 2D stylized images for downstream 3D reconstruction?
• RQ4 Can optimized latent code offsets effectively isolate expression changes from other semantic attributes like identity and hair color?
• RQ5 How well do the reconstructed 3D shapes generalize across viewpoints and lighting conditions, and how accurate is the surface normal estimation?

Key findings

• The method successfully generates high-fidelity 3D cartoon face shapes from a single 2D GAN-generated input image without any 3D supervision.
• The optimized latent code offsets enable expression manipulation while preserving identity and shape, reducing unintended semantic changes compared to baseline SeFa.
• The reconstructed 3D meshes preserve fine-grained details such as wrinkles, skin color, and hair texture, as shown in qualitative comparisons.
• Relighting and multi-view rendering results demonstrate accurate surface normal maps, with lighting changes preserving original facial features.
• The framework achieves consistent alignment between 2D human faces, stylized cartoon images, and 3D shapes through shared latent space representation.
• Limitations include unnatural distortions at large viewpoint rotations and residual changes in hair color during expression editing, indicating challenges in perfect disentanglement.