[论文解读] On Measuring and Controlling the Spectral Bias of the Deep Image Prior

Zenglin Shi, Pascal Mettes+2 more|arXiv (Cornell University)|Jul 2, 2021

Photoacoustic and Ultrasonic Imaging被引用 5

一句话总结

本文提出一种频带对应度量方法，用于衡量和控制深度图像先验中的谱偏差，即低频分量比高频分量学习得更快更好。通过引入Lipschitz约束卷积层和高斯控制上采样层，该方法在优化过程中防止性能下降，并实现自动停止，无需使用oracle停止标准，在图像去噪、去块效应、图像修复、超分辨率和细节增强等任务中均取得当前最优结果。

ABSTRACT

The deep image prior showed that a randomly initialized network with a suitable architecture can be trained to solve inverse imaging problems by simply optimizing it's parameters to reconstruct a single degraded image. However, it suffers from two practical limitations. First, it remains unclear how to control the prior beyond the choice of the network architecture. Second, training requires an oracle stopping criterion as during the optimization the performance degrades after reaching an optimum value. To address these challenges we introduce a frequency-band correspondence measure to characterize the spectral bias of the deep image prior, where low-frequency image signals are learned faster and better than high-frequency counterparts. Based on our observations, we propose techniques to prevent the eventual performance degradation and accelerate convergence. We introduce a Lipschitz-controlled convolution layer and a Gaussian-controlled upsampling layer as plug-in replacements for layers used in the deep architectures. The experiments show that with these changes the performance does not degrade during optimization, relieving us from the need for an oracle stopping criterion. We further outline a stopping criterion to avoid superfluous computation. Finally, we show that our approach obtains favorable results compared to current approaches across various denoising, deblocking, inpainting, super-resolution and detail enhancement tasks. Code is available at \url{https://github.com/shizenglin/Measure-and-Control-Spectral-Bias}.

研究动机与目标

• 理解并量化深度图像先验中的谱偏差，即低频图像分量比高频分量更容易被学习。
• 解决除网络架构选择外对深度图像先验缺乏控制的问题。
• 通过防止对高频噪声的过拟合，避免优化过程中的性能下降。
• 开发一种可靠且自动的停止准则，避免不必要的计算。
• 在多种逆向成像任务中提升收敛速度和重建质量。

提出的方法

• 提出频带对应度量，通过比较优化过程中不同频带的学习速率，定量衡量谱偏差。
• 提出Lipschitz约束卷积层，通过限制梯度流动来限制高频拟合，稳定训练过程。
• 引入高斯控制上采样层，允许可调平滑度以平衡频率响应并加速收敛。
• 采用带Lipschitz约束的改进批量归一化层，进一步稳定优化并提升泛化能力。
• 基于谱偏差监测开发自动停止准则，在无需oracle监督的情况下于性能峰值时停止优化。
• 在固定5,000次迭代下使用可调超参数（如式5中的λ）控制图像增强任务中的平滑度。

Figure 1: Frequency-band correspondence metric. The left image shows an example of correspondence map $H$ , which is computed according to Eq. ( 1 ). We divide the correspondence map into $N$ subgroups corresponding to $N$ non-overlapping frequency bands. Since the correspondence map is symmetrical

实验结果

研究问题

• RQ1 谱偏差在深度图像先验优化过程中如何表现？是否可被定量测量？
• RQ2 能否通过架构修改控制谱偏差，以防止性能下降？
• RQ3 能否使深度图像先验对高频噪声过拟合具有鲁棒性，而无需依赖oracle停止？
• RQ4 能否基于谱行为设计自动停止准则，以减少计算浪费？
• RQ5 控制谱偏差是否能提升在多样化逆向成像任务中的性能？

主要发现

• 在Set14数据集上，该方法在图像去噪任务中达到32.76的PSNR，优于Ulyanov等人（2020）和LapSRN等先前方法。
• 在8倍超分任务中，该方法达到30.45的PSNR，超过LapSRN的先前最先进水平28.54。
• 该方法在优化过程中完全消除了性能下降，维持峰值性能，无需oracle停止。
• 自动停止准则有效减少了不必要的计算，同时保持高质量结果。
• 在图像增强任务中，调节λ超参数可精确控制锐化强度，实现一致的细节增强。
• 在高噪声水平（σ=100）下出现失败，原因在于低频处噪声与自然图像的功率谱重叠，限制了谱分离。

Figure 2: Network architectures used in the experiments of Section 3 . The Encoder-Decoder is the same as the one used in Ulyanov et al. ( 2020 ) . Specifically, the encoder contains five convolution blocks. Each block contains two convolution layers with the kernel size of $3\times 3$ and the chann