[Paper Review] Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining

Yiqun Mei, Yuchen Fan+4 more|arXiv (Cornell University)|Jun 2, 2020

Advanced Image Processing TechniquesComputer Science48 references42 citations

TL;DR

The paper proposes a cross-scale non-local attention mechanism that preserves high-frequency details for image super-resolution, outperforming naïve cross-scale and in-scale approaches. It demonstrates superior PSNR on Set14 (×2) and qualitative improvements on Urban100 and Manga109.

ABSTRACT

Deep convolution-based single image super-resolution (SISR) networks embrace the benefits of learning from large-scale external image resources for local recovery, yet most existing works have ignored the long-range feature-wise similarities in natural images. Some recent works have successfully leveraged this intrinsic feature correlation by exploring non-local attention modules. However, none of the current deep models have studied another inherent property of images: cross-scale feature correlation. In this paper, we propose the first Cross-Scale Non-Local (CS-NL) attention module with integration into a recurrent neural network. By combining the new CS-NL prior with local and in-scale non-local priors in a powerful recurrent fusion cell, we can find more cross-scale feature correlations within a single low-resolution (LR) image. The performance of SISR is significantly improved by exhaustively integrating all possible priors. Extensive experiments demonstrate the effectiveness of the proposed CS-NL module by setting new state-of-the-arts on multiple SISR benchmarks.

Motivation & Objective

• Motivate improved super-resolution by leveraging cross-scale self-exemplars without down-sampling high-frequency information.
• Introduce a cross-scale non-local attention mechanism that maintains high-resolution features.
• Show quantitative and qualitative improvements over naïve cross-scale and standard in-scale attention methods.

Proposed method

• Define cross-scale non-local attention that links target-scale features with corresponding high-resolution self-exemplars without down-sampling.
• Formulate attention as a non-local sum over spatial positions with scale-aware feature pairs (Eq. 3).
• Compare with naïve cross-scale attention (Eq. 2) and in-scale attention (Eq. 1) to demonstrate benefits of preserving high-frequency details.
• Provide experimental results on standard SR benchmarks (Set14, Urban100, Manga109) including PSNR/visual comparisons.

Experimental results

Research questions

• RQ1 Does cross-scale non-local attention that preserves high-frequency information outperform naïve cross-scale and in-scale approaches for SR?
• RQ2 How does the proposed CSNLN affect quantitative metrics (e.g., PSNR) and qualitative detail reconstruction on diverse datasets?

Key findings

• Cross-scale non-local attention (CSNLN) yields higher PSNR on Set14 ×2 (33.74) than naïve cross-scale (33.65) and in-scale (33.62).
• Naïve cross-scale attention provides negligible gains over in-scale attention, highlighting the importance of preserving high-frequency details.
• Qualitative results show finer reconstructed details on Urban100 and Manga109 compared to several state-of-the-art SR methods.
• CSNLN demonstrates improvements in reconstructing high-frequency structures due to maintaining high-resolution feature maps and self-exemplar hints.