[Paper Review] Efficient Image Super-Resolution Using Pixel Attention

Hengyuan Zhao, Xiangtao Kong+3 more|arXiv (Cornell University)|Oct 2, 2020

Advanced Image Processing Techniques46 references43 citations

TL;DR

PAN introduces pixel attention (PA) to build a lightweight SR network and achieves competitive PSNR with far fewer parameters than baseline lightweight models. It includes SC-PA blocks for the main branch and U-PA blocks for reconstruction.

ABSTRACT

This work aims at designing a lightweight convolutional neural network for image super resolution (SR). With simplicity bare in mind, we construct a pretty concise and effective network with a newly proposed pixel attention scheme. Pixel attention (PA) is similar as channel attention and spatial attention in formulation. The difference is that PA produces 3D attention maps instead of a 1D attention vector or a 2D map. This attention scheme introduces fewer additional parameters but generates better SR results. On the basis of PA, we propose two building blocks for the main branch and the reconstruction branch, respectively. The first one - SC-PA block has the same structure as the Self-Calibrated convolution but with our PA layer. This block is much more efficient than conventional residual/dense blocks, for its twobranch architecture and attention scheme. While the second one - UPA block combines the nearest-neighbor upsampling, convolution and PA layers. It improves the final reconstruction quality with little parameter cost. Our final model- PAN could achieve similar performance as the lightweight networks - SRResNet and CARN, but with only 272K parameters (17.92% of SRResNet and 17.09% of CARN). The effectiveness of each proposed component is also validated by ablation study. The code is available at https://github.com/zhaohengyuan1/PAN.

Motivation & Objective

• Motivate efficient single-image super-resolution (SISR) with a lightweight model.
• Propose pixel attention (PA) that generates 3D attention maps with minimal parameters.
• Integrate PA into Self-Calibrated convolution blocks (SC-PA) for the main path.
• Introduce a reconstruction block (U-PA) that applies PA during upsampling with nearest-neighbor interpolation.
• Demonstrate competitive performance against lightweight SR models while drastically reducing parameter count.

Proposed method

• Introduce Pixel Attention (PA) as a 3D attention map produced by a 1x1 conv + sigmoid.
• Embed PA into the Self-Calibrated convolution framework to form SC-PA blocks (two-branch structure).
• Use PA in the reconstruction/up-sampling branch via the U-PA block (NN upsampling, conv, PA, conv).
• Construct the PAN network with three modules: feature extraction, nonlinear mapping (stacked SC-PA blocks), and reconstruction (U-PA blocks).
• Use a global skip path with bilinear upsampling of the LR input for final SR fusion: I_SR = rec(x_n) + up(I_LR).
• Train with L1 loss, Adam optimizer, cosine annealing learning rate schedule, using DIV2K/Flickr2K as training data and standard 5-benchmark evaluation.

Experimental results

Research questions

• RQ1 Can a lightweight SR network with a minimal parameter budget (≈272K) match or exceed performance of existing lightweight models like SRResNet and CARN?
• RQ2 Does pixel attention (PA) provide a more effective attention mechanism for SR than channel or spatial attention in small networks?
• RQ3 What is the impact of integrating PA into the main SC block versus reconstruction upsampling stages?
• RQ4 How does the proposed PAN balance performance versus parameter count across standard SR benchmarks and scaling factors (×2, ×3, ×4)?

Key findings

• PAN attains competitive PSNR/SSIM while having far fewer parameters (~272K) than SRResNet (~1.37M) and CARN (~1.59M).
• Pixel Attention (PA) outperforms channel attention and spatial attention in ablations for lightweight networks.
• SC-PA (Self-Calibrated with PA) improves PSNR over baseline RB-PA, with SC-PA yielding notable gains at modest parameter cost.
• PA in both SC-PA and U-PA contributes to better SR performance, with measured gains in PSNR on benchmark tests.
• PA is especially beneficial for small models and may be less impactful for much larger networks, where training strategies and redundancy play larger roles.