[论文解读] Cross-Scale Internal Graph Neural Network for Image Super-Resolution

Shangchen Zhou, Jiawei Zhang+2 more|arXiv (Cornell University)|Jun 30, 2020

Advanced Image Processing Techniques参考文献 46被引用 116

一句话总结

本文提出 IGNN，一种跨尺度内部图神经网络，利用跨尺度补丁重现将 HR 线索传递给 LR 补丁，以提升单图像超分辨率。

ABSTRACT

Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. However, for single image super-resolution (SISR), most existing deep non-local methods (e.g., non-local neural networks) only exploit similar patches within the same scale of the low-resolution (LR) input image. Consequently, the restoration is limited to using the same-scale information while neglecting potential high-resolution (HR) cues from other scales. In this paper, we explore the cross-scale patch recurrence property of a natural image, i.e., similar patches tend to recur many times across different scales. This is achieved using a novel cross-scale internal graph neural network (IGNN). Specifically, we dynamically construct a cross-scale graph by searching k-nearest neighboring patches in the downsampled LR image for each query patch in the LR image. We then obtain the corresponding k HR neighboring patches in the LR image and aggregate them adaptively in accordance to the edge label of the constructed graph. In this way, the HR information can be passed from k HR neighboring patches to the LR query patch to help it recover more detailed textures. Besides, these internal image-specific LR/HR exemplars are also significant complements to the external information learned from the training dataset. Extensive experiments demonstrate the effectiveness of IGNN against the state-of-the-art SISR methods including existing non-local networks on standard benchmarks.

研究动机与目标

• Motivate leveraging internal, image-specific cross-scale patch recurrence for SISR to recover finer textures.
• Propose a cross-scale graph-based module (GraphAgg) to aggregate HR patches from downsampled references.
• Integrate GraphAgg into an IGNN backbone to pass HR information from cross-scale neighbors to LR queries.
• Show that internal cross-scale cues complement external training data and improve SR performance on standard benchmarks.

提出的方法

• Dynamically construct a cross-scale graph Gk by finding k nearest neighbors of each LR patch in the downsampled LR image.
• Aggregate k HR patches conditioned on edge labels via an edge-conditioned network (ECN) to produce enhanced LR features F'L.
• Use AdaPN to align low-frequency statistics between LR and HR neighboring patches before aggregation.
• Embed the aggregated HR features to match the LR resolution and fuse with LR features through skip connections.
• Backbone IGNN is built on EDSR; GraphAgg is inserted after the 16th residual block and uses small ECN/DEN sub-networks.

实验结果

研究问题

• RQ1 Can cross-scale recurrent patches provide meaningful high-resolution cues to improve LR-to-HR reconstruction in SISR?
• RQ2 Does a graph-based cross-scale aggregation outperform same-scale non-local or KNN-based methods for SR?
• RQ3 How do AdaPN and ECN components affect the quality and robustness of cross-scale patch aggregation?
• RQ4 Where in the backbone is cross-scale GraphAgg most beneficial for SR performance?
• RQ5 What is the impact of key hyperparameters (k, window size d) on IGNN performance and efficiency?

主要发现

方法	尺度	Set5 PSNR	Set5 SSIM	Set14 PSNR	Set14 SSIM	BSD100 PSNR	BSD100 SSIM	Urban100 PSNR	Urban100 SSIM	Manga109 PSNR	Manga109 SSIM
IGNN (Ours)	x2	38.24	0.9613	34.07	0.9217	32.41	0.9025	33.23	0.9383	39.35	0.9786
IGNN+ (Ours)	x2	38.31	0.9616	34.18	0.9222	32.46	0.9030	33.42	0.9396	39.54	0.9790
IGNN (Ours)	x3	34.72	0.9298	30.66	0.8484	29.31	0.8105	29.03	0.8696	34.39	0.9496
IGNN+ (Ours)	x3	34.84	0.9305	30.75	0.8496	29.37	0.8115	29.20	0.8721	34.67	0.9509
IGNN (Ours)	x4	32.57	0.8998	28.85	0.7891	27.77	0.7434	26.84	0.8090	31.28	0.9182
IGNN+ (Ours)	x4	32.71	0.9011	28.96	0.7908	27.84	0.7447	27.04	0.8128	31.59	0.9207

• IGNN outperforms state-of-the-art CNN-based SR methods and existing non-local networks on standard benchmarks.
• Cross-scale GraphAgg provides notable SR gains by leveraging HR cues from downsampled references, not just internal LR similarities.
• Adaptive Patch Normalization and Edge-Conditioned sub-network significantly improve aggregation robustness, reducing color/brightness mismatches.
• GraphAgg placed in the middle of the backbone (after the 16th block) yields the largest performance gain.
• Self-ensemble IGNN+ further improves PSNR/SSIM across scales.