[Paper Review] High-Fidelity and Arbitrary Face Editing

Yue Gao, Fangyun Wei+5 more|arXiv (Cornell University)|Mar 29, 2021

Face recognition and analysis4 citations

TL;DR

HifaFace proposes a wavelet-based face editing framework that enhances high-fidelity and arbitrary facial attribute editing by directly injecting high-frequency image details into the generator and using a high-frequency discriminator to prevent steganographic signal hiding. The method achieves superior fidelity and controllability, outperforming SOTA approaches in both qualitative and quantitative evaluations.

ABSTRACT

Cycle consistency is widely used for face editing. However, we observe that the generator tends to find a tricky way to hide information from the original image to satisfy the constraint of cycle consistency, making it impossible to maintain the rich details (e.g., wrinkles and moles) of non-editing areas. In this work, we propose a simple yet effective method named HifaFace to address the above-mentioned problem from two perspectives. First, we relieve the pressure of the generator to synthesize rich details by directly feeding the high-frequency information of the input image into the end of the generator. Second, we adopt an additional discriminator to encourage the generator to synthesize rich details. Specifically, we apply wavelet transformation to transform the image into multi-frequency domains, among which the high-frequency parts can be used to recover the rich details. We also notice that a fine-grained and wider-range control for the attribute is of great importance for face editing. To achieve this goal, we propose a novel attribute regression loss. Powered by the proposed framework, we achieve high-fidelity and arbitrary face editing, outperforming other state-of-the-art approaches.

Motivation & Objective

• To address the issue of poor detail preservation in cycle-consistent face editing, where generators hide input details via steganographic signals.
• To improve high-fidelity face editing by directly feeding high-frequency image components into the generator to reduce reliance on hidden signal encoding.
• To enable fine-grained, wide-range control over facial attributes through a novel attribute regression loss.
• To enhance model generalization and fidelity using semi-supervised learning with large-scale unlabeled data.
• To re-evaluate and resolve the limitations of cycle consistency in GAN-based face editing.

Proposed method

• The method employs wavelet transformation to decompose input images into multi-frequency subbands, isolating high-frequency components that contain fine details like wrinkles and moles.
• A novel wavelet-based skip-connection module replaces standard skip-connections, directly injecting high-frequency features into the generator’s final layers to preserve detail fidelity.
• A high-frequency discriminator ($D_H$) is introduced to enforce the generator to produce realistic high-frequency details, discouraging steganographic signal hiding.
• An attribute regression loss is designed to explicitly model attribute changes across a continuous range, enabling smooth and arbitrary interpolation and extrapolation of attributes.
• Semi-supervised learning is applied to leverage large amounts of unlabeled face images, improving generalization and fidelity in wild image settings.
• The overall framework combines cycle consistency with explicit detail supervision and attribute control, forming a robust face editing pipeline.

(a) Input $\bm{x}$

Experimental results

Research questions

• RQ1 Can cycle-consistent GANs in face editing be improved by preventing generators from hiding input details through steganographic means?
• RQ2 How can high-frequency image components be effectively injected into the generator to preserve fine facial details like wrinkles and moles?
• RQ3 Can a high-frequency discriminator effectively enforce realistic detail synthesis and prevent trivial solutions to cycle consistency?
• RQ4 To what extent can a novel attribute regression loss enable fine-grained, continuous, and arbitrary control over facial attributes?
• RQ5 Does semi-supervised learning with large-scale unlabeled data significantly improve the quality and robustness of face editing in wild image settings?

Key findings

• The full HifaFace model achieves a Fréchet Inception Distance (FID) of 4.04 and a Fréchet Appearance Distance (FAD) of 97.5, significantly outperforming baseline methods.
• Ablation studies show that removing the wavelet-based skip-connection or the high-frequency discriminator leads to a sharp drop in performance, with FID rising to 5.34 and FAD falling to 96.4.
• The model without the attribute regression loss fails to produce smooth interpolation and extrapolation, especially for attributes like eyeglasses and aging, which require wide-range control.
• Semi-supervised learning improves synthesis quality, particularly for low-resource attributes like 'eyeglasses' and 'old', where labeled data are scarce.
• The model successfully edits wild images under diverse poses and expressions, maintaining high fidelity and detail preservation, as shown in Figure 10.
• The generator in HifaFace does not encode hidden signals in output images, as confirmed by reconstruction experiments, unlike baseline models that rely on steganography.

(b) $\bm{\hat{y}}=\bm{x}\text{ + EG}$