[论文解读] One-Shot Generative Domain Adaptation

Ceyuan Yang, Yujun Shen+5 more|arXiv (Cornell University)|Nov 18, 2021

Generative Adversarial Networks and Image Synthesis参考文献 34被引用 8

一句话总结

该论文提出GenDA，一种一次性生成式域自适应方法，仅使用一张参考图像即可将预训练的GAN迁移至新域。通过冻结原始生成器和判别器，并引入轻量级属性适配器与分类器，GenDA在有效迁移目标域特有属性的同时，保持了生成质量和多样性，相较于先前方法在极端数据稀缺条件下展现出更优的多样性与保真度。

ABSTRACT

This work aims at transferring a Generative Adversarial Network (GAN) pre-trained on one image domain to a new domain referring to as few as just one target image. The main challenge is that, under limited supervision, it is extremely difficult to synthesize photo-realistic and highly diverse images, while acquiring representative characters of the target. Different from existing approaches that adopt the vanilla fine-tuning strategy, we import two lightweight modules to the generator and the discriminator respectively. Concretely, we introduce an attribute adaptor into the generator yet freeze its original parameters, through which it can reuse the prior knowledge to the most extent and hence maintain the synthesis quality and diversity. We then equip the well-learned discriminator backbone with an attribute classifier to ensure that the generator captures the appropriate characters from the reference. Furthermore, considering the poor diversity of the training data (i.e., as few as only one image), we propose to also constrain the diversity of the generative domain in the training process, alleviating the optimization difficulty. Our approach brings appealing results under various settings, substantially surpassing state-of-the-art alternatives, especially in terms of synthesis diversity. Noticeably, our method works well even with large domain gaps, and robustly converges within a few minutes for each experiment.

研究动机与目标

• 为解决在仅有一张参考图像的情况下，将预训练GAN适配至新域时生成多样化、高质量图像的挑战。
• 克服在单次设置下微调GAN时常见的多样性崩溃问题。
• 在迁移目标图像最具代表性的属性的同时，保留源域GAN的丰富先验知识。
• 通过在单一样本条件下约束潜在空间多样性，降低优化难度。

提出的方法

• 引入轻量级属性适配器模块，将潜在空间与生成器连接，该模块在冻结原始生成器权重的前提下进行训练，以保留先验知识。
• 在冻结主干网络的预训练判别器之上增加轻量级属性分类器，以引导生成器捕捉目标域特有属性。
• 在自适应过程中冻结整个生成器与判别器网络，仅优化新增模块，以维持生成质量与多样性。
• 在训练期间对潜在分布应用截断（truncation），以减少模式崩溃并提升低数据条件下的优化稳定性。
• 采用双流训练策略，其中生成器学习生成匹配目标域的图像，而判别器则基于域特定属性区分真实与生成图像。
• 使用FID、精确率与召回率等标准评估指标，在多个基准上验证性能。

Figure 1: Diagram of one-shot generative domain adaptation. Left: The overall framework, where a GAN model pre-trained on the large-scale source data is transferred to the target domain with only one training sample . A lightweight attribute adaptor and attribute classifier are introduced to the fro

实验结果

研究问题

• RQ1 能否仅使用一张参考图像，有效适配预训练GAN至新域，同时保持高生成多样性？
• RQ2 为何标准微调策略在单次域自适应中无法保留多样性？
• RQ3 如何复用源域GAN的先验知识，以提升域自适应中的数据效率？
• RQ4 潜在空间截断在最小监督条件下对训练稳定性有何作用？
• RQ5 能否使用轻量级、模块化组件仅迁移目标图像的关键属性，而无需重新训练整个GAN？

主要发现

• GenDA在单次生成式域自适应任务中达到最先进性能，FID与精确率-召回率指标显著优于现有方法。
• 即使仅使用一张训练图像，该方法仍能保持高生成多样性，避免了微调GAN中常见的模式崩溃现象。
• 方法在每项实验中仅需数分钟即可收敛，展现出即使在极低监督条件下仍具强大训练效率。
• 训练期间使用截断潜在空间可降低优化难度并提升生成样本质量。
• GenDA成功仅凭一张参考图像，将预训练的StyleGAN2模型适配至人脸与教堂等新域，生成真实且多样的样本。
• 消融实验证实，冻结原始GAN权重并仅使用属性适配器与分类器，相比全量微调可获得更优性能。

Figure 2: Qualitative comparison on one-shot adaptation between FreezeD (Mo et al., 2020 ) , Cross-Domain (Ojha et al., 2021 ) , and our proposed GenDA. The first column shows the reference images. GenDA significantly outperforms the other competitors from the diversity perspective.