[论文解读] Cross-Scale Internal Graph Neural Network for Image Super-Resolution

Shangchen Zhou, Jiawei Zhang+2 more|arXiv (Cornell University)|Jun 30, 2020

Advanced Image Processing Techniques参考文献 46被引用 116

一句话总结

本文提出 IGNN，一种跨尺度内部图神经网络，利用跨尺度补丁重现将 HR 线索传递给 LR 补丁，以提升单图像超分辨率。

ABSTRACT

Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. However, for single image super-resolution (SISR), most existing deep non-local methods (e.g., non-local neural networks) only exploit similar patches within the same scale of the low-resolution (LR) input image. Consequently, the restoration is limited to using the same-scale information while neglecting potential high-resolution (HR) cues from other scales. In this paper, we explore the cross-scale patch recurrence property of a natural image, i.e., similar patches tend to recur many times across different scales. This is achieved using a novel cross-scale internal graph neural network (IGNN). Specifically, we dynamically construct a cross-scale graph by searching k-nearest neighboring patches in the downsampled LR image for each query patch in the LR image. We then obtain the corresponding k HR neighboring patches in the LR image and aggregate them adaptively in accordance to the edge label of the constructed graph. In this way, the HR information can be passed from k HR neighboring patches to the LR query patch to help it recover more detailed textures. Besides, these internal image-specific LR/HR exemplars are also significant complements to the external information learned from the training dataset. Extensive experiments demonstrate the effectiveness of IGNN against the state-of-the-art SISR methods including existing non-local networks on standard benchmarks.

研究动机与目标

• Motivate leveraging internal, image-specific cross-scale patch recurrence for SISR to recover finer textures.
• Propose a cross-scale graph-based module (GraphAgg) to aggregate HR patches from downsampled references.
• Integrate GraphAgg into an IGNN backbone to pass HR information from cross-scale neighbors to LR queries.
• Show that internal cross-scale cues complement external training data and improve SR performance on standard benchmarks.

提出的方法

• Dynamically construct a cross-scale graph Gk by finding k nearest neighbors of each LR patch in the downsampled LR image.
• Aggregate k HR patches conditioned on edge labels via an edge-conditioned network (ECN) to produce enhanced LR features F'L.
• Use AdaPN to align low-frequency statistics between LR and HR neighboring patches before aggregation.
• Embed the aggregated HR features to match the LR resolution and fuse with LR features through skip connections.
• Backbone IGNN is built on EDSR; GraphAgg is inserted after the 16th residual block and uses small ECN/DEN sub-networks.

实验结果

研究问题

• RQ1 Can cross-scale recurrent patches provide meaningful high-resolution cues to improve LR-to-HR reconstruction in SISR?
• RQ2 Does a graph-based cross-scale aggregation outperform same-scale non-local or KNN-based methods for SR?
• RQ3 How do AdaPN and ECN components affect the quality and robustness of cross-scale patch aggregation?
• RQ4 Where in the backbone is cross-scale GraphAgg most beneficial for SR performance?
• RQ5 What is the impact of key hyperparameters (k, window size d) on IGNN performance and efficiency?

主要发现

• IGNN outperforms state-of-the-art CNN-based SR methods and existing non-local networks on standard benchmarks.
• Cross-scale GraphAgg provides notable SR gains by leveraging HR cues from downsampled references, not just internal LR similarities.
• Adaptive Patch Normalization and Edge-Conditioned sub-network significantly improve aggregation robustness, reducing color/brightness mismatches.
• GraphAgg placed in the middle of the backbone (after the 16th block) yields the largest performance gain.
• Self-ensemble IGNN+ further improves PSNR/SSIM across scales.