Skip to main content
Nubint
QUICK REVIEW

[论文解读] Unbalanced Feature Transport for Exemplar-based Image Translation

Fangneng Zhan, Yingchen Yu|arXiv (Cornell University)|Jun 19, 2021
Generative Adversarial Networks and Image Synthesis被引用 7
一句话总结

该论文提出UNITE，一种新颖的基于样本的图像转换框架，利用非平衡最优传输（UOT）对条件输入与风格样本之间的特征进行对齐，实现高保真、逼真的图像生成，并保持忠实的风格迁移。通过引入自适应质量学习和多阶段特征传输，UNITE缓解了多对一匹配问题，保留了细粒度纹理，在定性和定量指标上均优于当前最先进方法。

ABSTRACT

Despite the great success of GANs in images translation with different conditioned inputs such as semantic segmentation and edge maps, generating high-fidelity realistic images with reference styles remains a grand challenge in conditional image-to-image translation. This paper presents a general image translation framework that incorporates optimal transport for feature alignment between conditional inputs and style exemplars in image translation. The introduction of optimal transport mitigates the constraint of many-to-one feature matching significantly while building up accurate semantic correspondences between conditional inputs and exemplars. We design a novel unbalanced optimal transport to address the transport between features with deviational distributions which exists widely between conditional inputs and exemplars. In addition, we design a semantic-activation normalization scheme that injects style features of exemplars into the image translation process successfully. Extensive experiments over multiple image translation tasks show that our method achieves superior image translation qualitatively and quantitatively as compared with the state-of-the-art.

研究动机与目标

  • 为解决在使用参考样本进行条件图像到图像转换时的忠实风格控制挑战。
  • 克服基于余弦相似度的特征匹配方法常导致的多对一匹配及细节丢失问题。
  • 设计一种方法，有效对齐条件输入与样本之间分布差异较大的特征。
  • 通过多尺度、多阶段的特征传输，保留复杂纹理细节。
  • 开发一种风格注入机制，在图像生成过程中保持语义一致性。

提出的方法

  • 引入非平衡最优传输（UOT）以对齐条件输入与样本之间的特征，结合自适应质量学习以处理分布不匹配问题。
  • 采用多阶段特征传输策略，在翻译过程中保留不同尺度下的细节纹理。
  • 设计语义激活归一化（SEACE）层，将对齐后的风格特征注入翻译网络，同时保持语义一致性。
  • 端到端联合优化特征传输与翻译网络，以提升图像保真度与风格准确性。
  • 利用最优传输整体匹配特征，避免传统余弦相似度方法中逐特征匹配带来的缺陷。
  • 采用双流架构：一个特征传输网络用于对齐，一个翻译网络用于图像合成。
Figure 1: Different feature matching in image translation: Cosine Similarity tends to match each feature separately which often leads to many-to-one matching. Classical optimal transport ( Classical OT ) suppresses the many-to-one matching problem but it matches all feature points including undesire
Figure 1: Different feature matching in image translation: Cosine Similarity tends to match each feature separately which often leads to many-to-one matching. Classical optimal transport ( Classical OT ) suppresses the many-to-one matching problem but it matches all feature points including undesire

实验结果

研究问题

  • RQ1如何改进条件输入与样本之间的特征对齐，以减少多对一匹配并保留细粒度细节？
  • RQ2非平衡最优传输在处理条件输入与样本之间分布不匹配问题中起到什么作用？
  • RQ3多阶段特征传输如何促进图像翻译中复杂纹理的保留？
  • RQ4新型归一化方案能否有效注入风格特征，同时保持语义一致性？
  • RQ5所提方法在真实感与风格保真度方面，相较于现有基于样本的图像翻译方法，优势程度如何？

主要发现

  • UNITE 在 CelebA-HQ 数据集上取得 13.15 的 Fréchet Inception Distance（FID）分数，显著优于基线方法 SPADE（31.50）及其他 SOTA 方法。
  • 该方法将 Swin Distance（SWD）降低至 14.91，表明生成图像与真实图像之间的分布相似性得到提升。
  • UNITE 的 LPIPS 得分为 0.213，表明与真实图像具有高度感知相似性，用户偏好度达 30%。
  • 消融研究证实，结合自适应质量学习的 UOT 有效减少了错误匹配与多对一匹配，性能优于经典最优传输方法。
  • 多阶段传输与 SEACE 组件在定性消融研究中均显著促进纹理保留与风格一致性。
  • UNITE 在多个数据集（包括 CelebA-HQ 与 DeepFashion）上展现出更优的多样性与真实感，实现对样本的忠实风格迁移。
Figure 2: The framework of our proposed network: The Conditional Input and Exemplar are fed to feature extractors $F_{X}$ and $F_{Z}$ to extract feature vectors $X$ and $Z$ . The mass (or weight) of the feature vectors ( $\alpha$ and $\beta$ masses) are then determined collectively by $X$ and $Z$ .
Figure 2: The framework of our proposed network: The Conditional Input and Exemplar are fed to feature extractors $F_{X}$ and $F_{Z}$ to extract feature vectors $X$ and $Z$ . The mass (or weight) of the feature vectors ( $\alpha$ and $\beta$ masses) are then determined collectively by $X$ and $Z$ .

更好的研究，从现在开始

从阅读论文到最终审阅，大幅缩短您的研究时间。

无需绑定信用卡

本解读由 AI 生成，并经人工编辑审核。