Skip to main content
Nubint
QUICK REVIEW

[论文解读] Facial Attribute Transformers for Precise and Robust Makeup Transfer

Zhaoyi Wan, Hao Chen|arXiv (Cornell University)|Apr 7, 2021
Face recognition and analysis被引用 5
一句话总结

本文提出了一种名为面部属性变换器（FAT）和空间FAT的新框架，用于实现高保真、鲁棒的妆容迁移。该方法利用自注意力机制建模源图像与参考图像之间的语义对应关系，实现精确的颜色与纹理迁移，同时通过可微分的薄板样条（TPS）实现几何形状变换。该方法在颜色保真度、空间形变和对姿态及光照变化的鲁棒性方面均达到当前最优性能。

ABSTRACT

In this paper, we address the problem of makeup transfer, which aims at transplanting the makeup from the reference face to the source face while preserving the identity of the source. Existing makeup transfer methods have made notable progress in generating realistic makeup faces, but do not perform well in terms of color fidelity and spatial transformation. To tackle these issues, we propose a novel Facial Attribute Transformer (FAT) and its variant Spatial FAT for high-quality makeup transfer. Drawing inspirations from the Transformer in NLP, FAT is able to model the semantic correspondences and interactions between the source face and reference face, and then precisely estimate and transfer the facial attributes. To further facilitate shape deformation and transformation of facial parts, we also integrate thin plate splines (TPS) into FAT, thus creating Spatial FAT, which is the first method that can transfer geometric attributes in addition to color and texture. Extensive qualitative and quantitative experiments demonstrate the effectiveness and superiority of our proposed FATs in the following aspects: (1) ensuring high-fidelity color transfer; (2) allowing for geometric transformation of facial parts; (3) handling facial variations (such as poses and shadows) and (4) supporting high-resolution face generation.

研究动机与目标

  • 解决现有妆容迁移方法在颜色保真度和空间变换精度方面的局限性。
  • 实现在源图像与参考图像之间对颜色/纹理属性以及几何属性（如眉毛形状）的精确迁移。
  • 开发一种可泛化的框架，支持高分辨率人脸生成，并能处理姿态和阴影等面部变化。
  • 设计一种灵活的真值生成策略与后处理流程，以提升训练与推理质量。
  • 将方法的适用性扩展至其他面部属性迁移任务，如面部年龄或表情迁移。

提出的方法

  • 提出面部属性变换器（FAT），其自注意力机制受自然语言处理中Transformer的启发，用于建模源图像与参考图像之间的语义对应关系。
  • 将薄板样条（TPS）集成到FAT中，构建空间FAT，实现面部部件的可微分几何变换。
  • 采用循环一致性生成对抗网络（cycle-consistent GAN）框架，结合对抗损失、循环损失、感知损失和专用妆容损失进行训练。
  • 使用VGG-16特征提取器计算感知损失，以保留身份特征与面部细节。
  • 设计一种灵活的真值生成策略，通过伪迁移图像与混合技术指导模型训练。
  • 应用高分辨率后处理操作，生成细节丰富、保真度高的面部图像。
Figure 1: (Best viewed in color) Our Facial Attribute Transformer (FAT) can faithfully transfer detailed attributes (such as color and texture) from the reference to the source. Spatial FAT, an extension of FAT, further enables spatial transformation. The proposed facial attribute transformers exhib
Figure 1: (Best viewed in color) Our Facial Attribute Transformer (FAT) can faithfully transfer detailed attributes (such as color and texture) from the reference to the source. Spatial FAT, an extension of FAT, further enables spatial transformation. The proposed facial attribute transformers exhib

实验结果

研究问题

  • RQ1基于Transformer的架构能否提升源图像与参考图像之间语义对应关系的建模能力，从而实现更优的妆容迁移？
  • RQ2能否有效迁移如眉毛形状等几何属性，同时保持颜色与纹理的准确性？
  • RQ3将TPS与注意力机制结合，是否能提升在姿态与光照变化下的空间形变鲁棒性与精确性？
  • RQ4所提方法是否无需架构修改即可泛化至其他面部属性迁移任务？
  • RQ5在颜色保真度与空间精度方面，该方法与当前最优方法相比，在定量与定性指标上表现如何？

主要发现

  • 空间FAT在颜色保真度方面表现卓越，生成图像中嘴唇与眼睛的颜色在色相、饱和度与纹理上均与参考图像高度一致。
  • 该方法成功迁移了几何属性（如眉毛形状），而此前方法如BeautyGAN与PSGAN则无法保持此类属性不变。
  • 定量结果表明FID与LPIPS指标显著提升，表明生成图像具有更高的感知质量与更低的分布差异。
  • 模型对姿态与阴影等面部变化表现出强鲁棒性，在复杂场景下优于现有方法。
  • 通过专门的后处理步骤，成功实现了高分辨率人脸生成，同时保留了精细细节。
  • 消融实验表明，该框架可良好泛化至其他属性迁移任务，如面部年龄迁移。
Figure 2: The GAN training of makeup transfer. $G$ is short for the generator, and $D_{x}$ is the discriminator for reference. $L_{1}$ is the Manhattan distance that performs as a criterion. The generator output is marked with dash borders. Note the reference and source are symmetrically exchanged i
Figure 2: The GAN training of makeup transfer. $G$ is short for the generator, and $D_{x}$ is the discriminator for reference. $L_{1}$ is the Manhattan distance that performs as a criterion. The generator output is marked with dash borders. Note the reference and source are symmetrically exchanged i

更好的研究，从现在开始

从阅读论文到最终审阅，大幅缩短您的研究时间。

无需绑定信用卡

本解读由 AI 生成，并经人工编辑审核。