[论文解读] PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance

Peiqing Yang, Shangchen Zhou+2 more|arXiv (Cornell University)|Sep 19, 2023

Face recognition and analysis被引用 9

一句话总结

PGDiff 提出部分引导，通过高质量图像属性来引导扩散模型去噪，从而实现无需降解过程建模的多用途人脸恢复。

ABSTRACT

Exploiting pre-trained diffusion models for restoration has recently become a favored alternative to the traditional task-specific training approach. Previous works have achieved noteworthy success by limiting the solution space using explicit degradation models. However, these methods often fall short when faced with complex degradations as they generally cannot be precisely modeled. In this paper, we propose PGDiff by introducing partial guidance, a fresh perspective that is more adaptable to real-world degradations compared to existing works. Rather than specifically defining the degradation process, our approach models the desired properties, such as image structure and color statistics of high-quality images, and applies this guidance during the reverse diffusion process. These properties are readily available and make no assumptions about the degradation process. When combined with a diffusion prior, this partial guidance can deliver appealing results across a range of restoration tasks. Additionally, PGDiff can be extended to handle composite tasks by consolidating multiple high-quality image properties, achieved by integrating the guidance from respective tasks. Experimental results demonstrate that our method not only outperforms existing diffusion-prior-based approaches but also competes favorably with task-specific models.

研究动机与目标

• 在不假设特定降解过程的前提下，使用扩散模型推动恢复。
• 建模所需的高质量（HQ）图像属性，并利用这些属性引导扩散去噪。
• 通过叠加多个 HQ 属性的引导，实现组合任务。
• 展示相对于基于扩散先验的方法的优越性，以及在任务特定模型上的竞争力。

提出的方法

• 使用分类器引导，通过反向传播梯度将扩散去噪限制在 HQ 属性附近。
• 引入动态引导权重和每个去噪步骤的多次梯度步骤，以提高引导效果。
• 用分类器表示每个 HQ 属性，输出目标引导损失以用于引导。
• 通过对多个属性的损失求和来实现组合任务，以应对复杂降解。
• 可选地加入感知损失和对抗性损失以进一步提升质量。

Figure 1: Overview of Our PGDiff Framework for Versatile Face Restoration . Here, we take the colorization task as an example to illustrate our inference pipeline. One may refer to Table 1 for the corresponding details ( e.g. , property, classifier, and target) of other tasks. We show that our metho

实验结果

研究问题

• RQ1 在不进行显式降解建模的情况下，高质量图像属性能否引导基于扩散的恢复？
• RQ2 部分引导在同质和组合人脸修复任务中的表现如何？
• RQ3 动态引导和多步梯度引导能否提升输出质量和可控性？
• RQ4 如何将多个 HQ 属性组合以应对诸如老照片修复等复杂降解？
• RQ5 向 PGDiff 添加感知或对抗性引导的影响是什么？

主要发现

• 在具有挑战性的恢复任务中，PGDiff 的表现优于基于扩散先验的方法。
• 在多样化任务中，PGDiff 与任务特定模型相较具有竞争力。
• 带归一化权重的动态引导在更好地遵循目标属性方面优于传统引导。
• 每个去噪步骤的多次梯度步骤增强引导并降低伪影。
• 组合引导通过结合多种 HQ 属性实现对复杂降解的处理。
• 结合感知和对抗性损失可进一步提升恢复质量。

Figure 2: Comparison on Blind Face Restoration. Input faces are corrupted by real-world degradations. Our PGDiff produces high-quality faces with faithful details. ( Zoom in for best view. )