[Paper Review] Semi-parametric Makeup Transfer via Semantic-aware Correspondence

Mingrui Zhu, Yun Yi+3 more|arXiv (Cornell University)|Mar 4, 2022

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

This paper proposes a semi-parametric makeup transfer method, SpMT, which combines non-parametric semantic-aware correspondence (SaC) with a parametric decoder to transfer makeup styles from a reference image to a non-makeup source image. The SaC module establishes semantic-aware feature correspondences between source and reference images, enabling robust, flexible, and high-quality makeup transfer that preserves identity and structure while handling pose, expression, and occlusion variations.

ABSTRACT

The large discrepancy between the source non-makeup image and the reference makeup image is one of the key challenges in makeup transfer. Conventional approaches for makeup transfer either learn disentangled representation or perform pixel-wise correspondence in a parametric way between two images. We argue that non-parametric techniques have a high potential for addressing the pose, expression, and occlusion discrepancies. To this end, this paper proposes a extbf{S}emi- extbf{p}arametric extbf{M}akeup extbf{T}ransfer (SpMT) method, which combines the reciprocal strengths of non-parametric and parametric mechanisms. The non-parametric component is a novel extbf{S}emantic- extbf{a}ware extbf{C}orrespondence (SaC) module that explicitly reconstructs content representation with makeup representation under the strong constraint of component semantics. The reconstructed representation is desired to preserve the spatial and identity information of the source image while "wearing" the makeup of the reference image. The output image is synthesized via a parametric decoder that draws on the reconstructed representation. Extensive experiments demonstrate the superiority of our method in terms of visual quality, robustness, and flexibility. Code and pre-trained model are available at \url{https://github.com/AnonymScholar/SpMT.

Motivation & Objective

• Address the limitations of fully parametric makeup transfer models that fail to handle blemishes and lack realism in material application.
• Overcome challenges posed by pose, expression, and occlusion discrepancies in real-world makeup transfer scenarios.
• Leverage the strengths of non-parametric methods—such as direct material reuse from reference images—while retaining the efficiency of parametric decoding.
• Enable controllable and robust makeup transfer through a hybrid parametric-nonparametric framework.
• Improve visual quality and identity preservation by explicitly modeling component-level semantics during correspondence learning.

Proposed method

• Propose a novel Semantic-aware Correspondence (SaC) module that establishes cross-image feature correspondences at multiple levels of a feature pyramid.
• Use the SaC module to non-parametrically extract makeup material (e.g., lip color, foundation) from the reference image and transfer it to the source image based on semantic component alignment.
• Apply a cosine similarity-based matching strategy within the SaC module to ensure correspondence is guided by semantic meaning, not just spatial proximity.
• Introduce a cosmetic perceptual loss that enforces semantic consistency between source and reconstructed representations during training.
• Feed the reconstructed, makeup-augmented content representation into a parametric decoder to generate the final after-makeup image.
• Train the model using adversarial loss, cycle-consistency loss, and the proposed cosmetic perceptual loss to ensure realism and fidelity.

Experimental results

Research questions

• RQ1 Can a non-parametric mechanism improve robustness to pose, expression, and occlusion variations in makeup transfer compared to fully parametric models?
• RQ2 How does semantic-aware correspondence enhance the realism and accuracy of makeup material transfer compared to pixel-wise or non-semantic correspondence?
• RQ3 To what extent can the hybrid semi-parametric design enable controllable makeup transfer (e.g., interpolation, part-specific transfer) without requiring retraining?
• RQ4 Does the proposed method preserve identity and structural details of the source image better than state-of-the-art parametric approaches?
• RQ5 Can the method achieve superior visual quality and generalization on real-world, unconstrained images with environmental variations?

Key findings

• SpMT achieves the lowest Fréchet Inception Distance (FID) scores across three makeup styles—Western (141.3), vFG (156.4), and vHX (119.2)—indicating superior distributional similarity to reference makeup images.
• The method achieves an SSIM score of 0.89, significantly higher than BeautyGAN (0.87), PSGAN (0.80), and SCGAN (0.81), demonstrating superior content and identity preservation.
• In user studies, SpMT received the highest vote ratio (over 40%) on both well-aligned and challenging images with pose, expression, and occlusion variations.
• The method successfully avoids artifacts such as lipstick appearing inside the mouth, a common failure in prior methods like BeautyGAN and PSGAN.
• SpMT enables flexible, controllable makeup transfer through interpolation and part-specific selection of reconstructed representations, as demonstrated in qualitative results.
• Despite a computational cost of ~1 second per image on a single NVIDIA 3090 GPU, the method maintains high visual quality and robustness, with code and pre-trained models publicly available.