[论文解读] Learning Graph Regularisation for Guided Super-Resolution

Riccardo de Lutio, Alexander Becker+4 more|arXiv (Cornell University)|Mar 27, 2022

Advanced Optical Sensing Technologies被引用 6

一句话总结

本文提出了一种用于引导超分的可学习图正则化框架，通过结合深度特征提取与可微图优化，确保在上采样过程中对低分辨率源图像的精确保真度。通过从深度特征提取器学习边缘势能，并在稀疏像素网格上通过端到端可微优化层施加这些势能，该方法在重建精度和输出清晰度方面优于以往的深度学习与传统方法。

ABSTRACT

We introduce a novel formulation for guided super-resolution. Its core is a differentiable optimisation layer that operates on a learned affinity graph. The learned graph potentials make it possible to leverage rich contextual information from the guide image, while the explicit graph optimisation within the architecture guarantees rigorous fidelity of the high-resolution target to the low-resolution source. With the decision to employ the source as a constraint rather than only as an input to the prediction, our method differs from state-of-the-art deep architectures for guided super-resolution, which produce targets that, when downsampled, will only approximately reproduce the source. This is not only theoretically appealing, but also produces crisper, more natural-looking images. A key property of our method is that, although the graph connectivity is restricted to the pixel lattice, the associated edge potentials are learned with a deep feature extractor and can encode rich context information over large receptive fields. By taking advantage of the sparse graph connectivity, it becomes possible to propagate gradients through the optimisation layer and learn the edge potentials from data. We extensively evaluate our method on several datasets, and consistently outperform recent baselines in terms of quantitative reconstruction errors, while also delivering visually sharper outputs. Moreover, we demonstrate that our method generalises particularly well to new datasets not seen during training.

研究动机与目标

• 解决基于深度学习的引导超分方法在下采样重建时无法保证源图像保真度的局限性。
• 结合基于优化方法的鲁棒性与深度网络的上下文特征学习能力。
• 开发一种端到端可训练框架，其中图边势能从源图像和引导图像中学习，以正则化高分辨率目标重建。
• 通过显式优化而非隐式学习来强制结构保真度，从而在领域分布偏移下提升泛化能力。

提出的方法

• 引入一种可微优化层，对目标图像像素上的稀疏局部亲和图执行精确的最大后验概率推理。
• 通过应用于源图像和引导图像的深度卷积特征提取器预测图中的边势能，从而实现大感受野。
• 图结构固定为像素网格，但边权重从高层特征中学习，允许长距离的上下文传播。
• 通过优化层的反向传播，实现特征提取器与图正则化器的端到端训练。
• 该方法使用稀疏图以保持计算效率，同时确保优化过程中的梯度流动。
• 该框架应用于引导深度超分任务，在多个数据集和从×4到×16的上采样因子下进行评估。

Figure 1 : Our method takes as input a low-resolution source image and a high-resolution guide image of another modality to build a graph using high-level image features. The graph is then used in a differentiable optimisation layer as a regularisation to reconstruct the target .

实验结果

研究问题

• RQ1 可学习图正则化器是否能在保持高感知质量的同时提升引导超分中的重建保真度？
• RQ2 与传统的基于优化的方法及基于深度学习的方法相比，该方法在定量和定性性能上表现如何？
• RQ3 该方法在未见数据集和分布偏移下的泛化能力如何？
• RQ4 该方法对特征提取器架构的选择有多敏感？
• RQ5 在所提出的可微优化框架中，重建精度与推理速度之间的权衡如何？

主要发现

• 在NYUv2 ×8上采样任务中，该方法在所有评估方法中实现了最低的MAE，使用UResNet-50主干网络时MAE为1.42 cm。
• 在所有测试数据集和上采样因子下，该方法在MSE和MAE指标上均持续优于最先进基线方法。
• 该方法在训练期间未见过的新数据集上表现出稳健的泛化能力，显示出强大的领域泛化能力。
• 性能对主干网络选择不敏感，表明图正则化有效限制了特征提取器的表达能力。
• 使用原始颜色作为特征时性能显著下降（MAE = 2.30 cm），证实学习到的特征对有效正则化至关重要。
• 在RTX 2080 Ti上，每256²图像块的推理时间约为111 ms，虽慢于前馈模型，但能确保精确的源图像保真度。

Figure 2 : The architecture of our approach. A neural network backbone is employed to extract deep feature maps from the guide and source images. A graph over pixels is constructed based on pairwise affinities derived from these feature maps. Finally, a quadratic optimisation problem is solved for a