[Paper Review] Texture Deformation Based Generative Adversarial Networks for Face Editing

Wenting Chen, Xinpeng Xie+2 more|arXiv (Cornell University)|Dec 24, 2018

Generative Adversarial Networks and Image Synthesis24 references4 citations

TL;DR

This paper proposes TDB-GAN, a novel face editing framework that disentangles facial texture from deformation using a DAE-based encoder, then transfers target attributes (e.g., expression, gender, age) via texture-to-image translation before warping the texture back using spatial deformation. The method achieves superior visual quality, sharper details, and better identity preservation than existing GAN-based face editing models on CelebA and RaFD datasets.

ABSTRACT

Despite the significant success in image-to-image translation and latent representation based facial attribute editing and expression synthesis, the existing approaches still have limitations in the sharpness of details, distinct image translation and identity preservation. To address these issues, we propose a Texture Deformation Based GAN, namely TDB-GAN, to disentangle texture from original image and transfers domains based on the extracted texture. The approach utilizes the texture to transfer facial attributes and expressions without the consideration of the object pose. This leads to shaper details and more distinct visual effect of the synthesized faces. In addition, it brings the faster convergence during training. The effectiveness of the proposed method is validated through extensive ablation studies. We also evaluate our approach qualitatively and quantitatively on facial attribute and facial expression synthesis. The results on both the CelebA and RaFD datasets suggest that Texture Deformation Based GAN achieves better performance.

Motivation & Objective

• To address limitations in existing GAN-based face editing methods, such as poor detail sharpness, weak attribute transfer, and identity inconsistency.
• To disentangle facial texture from deformation and use texture as the primary medium for attribute transfer, reducing dependency on pose and shape.
• To improve training convergence and visual quality by leveraging texture-based image synthesis instead of direct image-to-image translation.
• To preserve identity during editing through a dedicated identity loss between input and generated images.
• To validate the superiority of texture-to-image translation over image-to-image translation in facial attribute and expression editing.

Proposed method

• The method uses a Deformable Autoencoder (DAE) to decompose input images into shading, albedo, and deformation components, then combines shading and albedo to generate a clean, aligned texture image.
• A GAN generator is conditioned on both the extracted texture and target domain labels (e.g., gender, expression) to synthesize a new texture with target attributes.
• The synthesized texture is warped back to the original spatial deformation to reconstruct the final face image, preserving geometric structure.
• An identity loss is applied between the input image and the generated image to enforce identity preservation during editing.
• The framework is trained using adversarial loss, L1 reconstruction loss, and identity loss to ensure realism and consistency.
• The approach enables multi-domain face editing (e.g., expression, gender, age) by learning disentangled representations in the texture space.

Experimental results

Research questions

• RQ1 Can disentangling texture from deformation improve the quality and realism of generated faces in face editing tasks?
• RQ2 Does texture-based image translation lead to better attribute transfer and sharper facial details compared to direct image-to-image translation?
• RQ3 Can identity preservation be significantly improved by introducing an identity loss in a texture-based GAN framework?
• RQ4 How does TDB-GAN perform in comparison to state-of-the-art models like StarGAN, CycleGAN, and IcGAN in both qualitative and quantitative evaluations?
• RQ5 Does the disentangled texture representation enhance the model's ability to synthesize distinct facial expressions?

Key findings

• TDB-GAN achieved the highest perceptual vote rate (57.33%) for smile transfer and 62.00% for age transfer in user studies, outperforming IcGAN, CycleGAN, and StarGAN.
• On the RaFD dataset, TDB-GAN-generated images achieved a facial expression classification accuracy of 97.28%, significantly higher than IcGAN (91.61%), CycleGAN (88.44%), and StarGAN (92.06%).
• The method produced sharper facial details, especially in eye regions, and more distinguishable expressions (e.g., angry, fearful) compared to StarGAN and other baselines.
• User studies confirmed that TDB-GAN generated images with better realism, attribute quality, and identity preservation than competing models.
• Ablation studies demonstrated that both the texture-based translation and identity loss are critical for performance, with the identity loss significantly improving identity consistency.
• TDB-GAN showed faster training convergence and better generalization across multiple domains due to the disentangled representation in the texture space.