[论文解读] Pivotal Tuning for Latent-based Editing of Real Images

Daniel Roich, Ron Mokady+2 more|arXiv (Cornell University)|Jun 10, 2021

Generative Adversarial Networks and Image Synthesis被引用 4

一句话总结

本文提出Pivotal Tuning Inversion（PTI）方法，通过在pivot潜在代码周围微调生成器，实现在StyleGAN中高保真、身份保持的面部图像编辑。通过手术式调整生成器以更好地表示分布外图像，PTI解决了失真与可编辑性之间的权衡，在具有挑战性的现实人脸图像上实现了最先进的定量指标与定性编辑效果。

ABSTRACT

Recently, a surge of advanced facial editing techniques have been proposed that leverage the generative power of a pre-trained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pre-trained generator's domain. As it turns out, however, StyleGAN's latent space induces an inherent tradeoff between distortion and editability, i.e. between maintaining the original appearance and convincingly altering some of its attributes. Practically, this means it is still challenging to apply ID-preserving facial latent-space editing to faces which are out of the generator's domain. In this paper, we present an approach to bridge this gap. Our technique slightly alters the generator, so that an out-of-domain image is faithfully mapped into an in-domain latent code. The key idea is pivotal tuning - a brief training process that preserves the editing quality of an in-domain latent region, while changing its portrayed identity and appearance. In Pivotal Tuning Inversion (PTI), an initial inverted latent code serves as a pivot, around which the generator is fined-tuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. This surgical training process ends up altering appearance features that represent mostly identity, without affecting editing capabilities. We validate our technique through inversion and editing metrics, and show preferable scores to state-of-the-art methods. We further qualitatively demonstrate our technique by applying advanced edits (such as pose, age, or expression) to numerous images of well-known and recognizable identities. Finally, we demonstrate resilience to harder cases, including heavy make-up, elaborate hairstyles and/or headwear, which otherwise could not have been successfully inverted and edited by state-of-the-art methods.

研究动机与目标

• 解决基于潜在空间的现实图像编辑中，分布外人脸因身份丢失或伪影导致的失真与可编辑性权衡问题。
• 实现在原始StyleGAN生成器分布外的现实图像上，高质量且身份保持的编辑。
• 开发一种个性化、高效的微调方法，在保持生成器可编辑能力的同时提升重建保真度。
• 在浓妆、复杂发型和头饰等具有挑战性的案例中展示鲁棒性。
• 为媒体制作和个性化图像处理等现实世界面部编辑应用提供实用框架。

提出的方法

• 该方法首先使用现成的技术将真实图像反演到生成器的潜在空间中，生成W+空间中的初始潜在代码。
• 从W空间中选择一个pivot潜在代码，作为生成器微调的锚点。
• 通过简短的优化过程（pivotal tuning）对生成器进行微调，以在局部潜在区域保持可编辑性的同时最小化失真。
• 通过正则化项强制实现局部身份保持，通过约束邻近潜在代码维持其身份，防止全局分布偏移。
• 微调过程通过结合L2和LPIPS重建损失以及身份正则化项，对生成器各层进行优化。
• 最终的生成器被个性化到输入图像，从而实现后续编辑时具有高保真度和最小失真。

Figure 2: An illustration of the PTI method. StyleGAN’s latent space is portrayed in two dimensions (see Tov et al. [ 38 ] ), where the warmer colors indicate higher densities of $W$ , i.e. regions of higher editability. On the left, we illustrate the generated samples before pivotal tuning. We can

实验结果

研究问题

• RQ1 我们能否在不牺牲可编辑性的前提下，有效减少基于潜在空间的分布外真实图像编辑中的重建失真？
• RQ2 局部化、生成器特定的微调过程能否在适应新身份的同时，保持StyleGAN原有的可编辑能力？
• RQ3 在具有挑战性的面部图像上，pivotal tuning与标准反演方法相比，在身份保持和视觉质量方面表现如何？
• RQ4 pivotal tuning能否在多个身份上实现低失真且一致的性能表现？
• RQ5 pivot潜在代码是否对初始化具有鲁棒性？能否在微调过程中固定以降低计算成本？

主要发现

• 与替代方法相比，PTI在微调前后比较反演代码时，LPIPS（0.015±5e−6）和MSE（0.0012±1e−6）得分显著更优，表明失真极小。
• 该方法在浓妆、复杂发型和头饰等具有挑战性的图像上实现了高质量编辑，而先前方法因反演伪影而失效。
• 消融研究显示，若用均值或随机潜在代码替代pivot代码，失真显著增加，证明了pivot选择的重要性。
• PTI保持了强大的可编辑性，例如在Serena Williams和Robert Downey Jr.等可识别人物上成功应用了微笑、姿态、年龄、表情等高级编辑。
• 完整两步流程在单张RTX 2080上耗时不足3分钟，其中pivotal tuning耗时不足2分钟，具备现实应用的实用性。
• 该方法对pivot代码初始化具有鲁棒性，优化pivot代码带来的性能增益可忽略不计，因此可将其固定以减少计算开销。

Figure 4: Reconstruction of out-of-domain samples. Our method (right) reconstructs out-of-domain visual details (left), such as face paintings or hands, significantly better than state-of-the-art methods (middle).