[论文解读] Iterative Prompt Learning for Unsupervised Backlit Image Enhancement

Zhexin Liang, Chongyi Li+3 more|arXiv (Cornell University)|Mar 30, 2023

Image Enhancement Techniques被引用 5

一句话总结

该论文提出CLIP-LIT，一种新颖的无监督背光图像增强方法，通过在CLIP潜在空间中进行迭代提示学习，利用对比语言-图像预训练（CLIP）来提升增强质量，且无需成对数据。通过在CLIP潜在空间中联合优化可学习提示和增强网络，该方法在多样化背光场景下实现了最先进的视觉质量和泛化性能，优于现有无监督和有监督方法。

ABSTRACT

We propose a novel unsupervised backlit image enhancement method, abbreviated as CLIP-LIT, by exploring the potential of Contrastive Language-Image Pre-Training (CLIP) for pixel-level image enhancement. We show that the open-world CLIP prior not only aids in distinguishing between backlit and well-lit images, but also in perceiving heterogeneous regions with different luminance, facilitating the optimization of the enhancement network. Unlike high-level and image manipulation tasks, directly applying CLIP to enhancement tasks is non-trivial, owing to the difficulty in finding accurate prompts. To solve this issue, we devise a prompt learning framework that first learns an initial prompt pair by constraining the text-image similarity between the prompt (negative/positive sample) and the corresponding image (backlit image/well-lit image) in the CLIP latent space. Then, we train the enhancement network based on the text-image similarity between the enhanced result and the initial prompt pair. To further improve the accuracy of the initial prompt pair, we iteratively fine-tune the prompt learning framework to reduce the distribution gaps between the backlit images, enhanced results, and well-lit images via rank learning, boosting the enhancement performance. Our method alternates between updating the prompt learning framework and enhancement network until visually pleasing results are achieved. Extensive experiments demonstrate that our method outperforms state-of-the-art methods in terms of visual quality and generalization ability, without requiring any paired data.

研究动机与目标

• 为解决无成对训练数据下无监督背光图像增强的挑战，该挑战限制了泛化能力并导致过增强或欠增强。
• 克服在复杂光照条件下为CLIP-based图像增强寻找准确固定提示的困难，因为不同光照条件下的表现差异显著。
• 通过学习动态、数据驱动的提示，提升CLIP的图文嵌入与像素级增强之间的对齐。
• 通过迭代优化提示和增强网络，结合排名学习以减少分布差距，从而提升鲁棒性和视觉质量。
• 证明CLIP的视觉-语言先验可被有效适配用于低层次图像恢复任务，而不仅限于分类或生成任务。

提出的方法

• 该方法通过最小化背光图像与负向提示在潜在空间中的CLIP嵌入距离，以及明亮图像与正向提示的距离，初始化提示。
• 使用CLIP-Enhance损失训练增强网络，该损失最大化增强图像与正向提示之间的相似性，同时最小化与负向提示的相似性。
• 通过交替进行提示嵌入微调和基于排名的损失函数更新增强网络，实现迭代优化，以减少背光图像、增强结果与明亮图像之间的分布差距。
• 通过排名学习更新提示学习框架，确保增强结果与正向提示的相似性高于与负向提示的相似性，也高于与明亮图像的相似性。
• 框架在无需真实标签监督的情况下，交替进行提示优化和增强网络训练，直至获得视觉上令人满意的输出。
• 该方法在CLIP嵌入空间中使用对比学习目标来引导增强过程，以更具可解释性和亮度敏感性的损失替代对抗性训练。

Figure 2: Motivation. CLIP scores of proper prompts demonstrate alignment with human annotations ( e.g . , well-lit images), suggesting that CLIP can serve as an indicator to differentiate between well-lit and backlit images. However, the best wordings could differ on a case-by-case basis due to com

实验结果

研究问题

• RQ1 CLIP的视觉-语言先验能否被有效用于无监督像素级图像增强，特别是背光图像？
• RQ2 在复杂光照场景下，可学习提示相比固定文本提示，能否提升CLIP-based图像分类的鲁棒性和准确性？
• RQ3 与固定提示或对抗性训练方法相比，迭代提示优化是否能进一步提升背光图像增强的泛化能力和视觉质量？
• RQ4 参考明亮图像的选择在多大程度上影响增强网络的性能？
• RQ5 在低层次图像恢复任务中，CLIP-based损失能否优于对抗性损失？

主要发现

• 在BAID测试集上，该方法实现了21.579的PSNR和0.883的SSIM，显著优于当前最先进方法。
• 与使用固定提示（背光/明亮）相比，使用可学习提示使PSNR提升6.831，SSIM提升0.060。
• 若移除排名学习损失（公式7和8），PSNR降至20.884，SSIM降至0.865，表明其在性能中起关键作用。
• 该方法获得55.682的MUSIQ分数，表明其具有出色的感知质量，并优于基于对抗性损失的训练方法（MUSIQ：52.416）。
• 用户研究表明，超过60%的参与者更偏好本方法的结果，优于所有基线方法，证实其具有更优的视觉质量。
• 该方法在多样化场景（人脸、风景、动物、建筑、夜间场景）中泛化良好，无需针对特定场景进行调优。

Figure 3: Visual comparison between our method and the state-of-the-art light enhancement methods, including exposure correction method (Afifi et al. [ 1 ] ), backlit enhancement method (ExCNet [ 32 ] ), low-light image enhancement methods (SCI [ 21 ] , Zero-DCE [ 9 ] , SNR-aware [ 29 ] , EnlightenG