[Paper Review] Attention in Attention Network for Image Super-Resolution

Haoyu Chen, Jinjin Gu+1 more|arXiv (Cornell University)|Apr 19, 2021

Advanced Image Processing Techniques41 references63 citations

TL;DR

The paper analyzes how attention modules affect SISR and introduces Attention in Attention Network (A2N) with a dynamic two-branch A2B block that adaptively weighs attention and non-attention paths to improve SR performance with modest parameter overhead.

ABSTRACT

Convolutional neural networks have allowed remarkable advances in single image super-resolution (SISR) over the last decade. Among recent advances in SISR, attention mechanisms are crucial for high-performance SR models. However, the attention mechanism remains unclear on why and how it works in SISR. In this work, we attempt to quantify and visualize attention mechanisms in SISR and show that not all attention modules are equally beneficial. We then propose attention in attention network (A$^2$N) for more efficient and accurate SISR. Specifically, A$^2$N consists of a non-attention branch and a coupling attention branch. A dynamic attention module is proposed to generate weights for these two branches to suppress unwanted attention adjustments dynamically, where the weights change adaptively according to the input features. This allows attention modules to specialize to beneficial examples without otherwise penalties and thus greatly improve the capacity of the attention network with few parameters overhead. Experimental results demonstrate that our final model A$^2$N could achieve superior trade-off performances comparing with state-of-the-art networks of similar sizes. Codes are available at https://github.com/haoyuc/A2N.

Motivation & Objective

• Quantify how attention modules influence SR across network stages.
• Demonstrate that not all attention modules are beneficial and identify when they help or hurt SR performance.
• Propose a low-cost dynamic two-branch attention mechanism (A2) and an A2N network to enhance SR.
• Show that A2N outperforms state-of-the-art lightweight SR models at similar sizes.

Proposed method

• Propose a dynamic attention framework that splits an attention block into a non-attention branch and an attention branch.
• Introduce a dynamic attention module that computes branch weights from the same input features with a sum-to-one constraint.
• Fuse branches via a learnable 1x1 convolution to form the next feature map.
• Build the A2 block (A2B) and stack them into an A2N network with shallow feature extraction and a reconstruction module.
• Evaluate on standard SR benchmarks with PSNR/SSIM on Y channel.
• Compare against multiple SR methods with similar parameter counts.

Experimental results

Research questions

• RQ1 Which parts of an image benefit most from attention mechanisms in SR, and how does this vary across network depth?
• RQ2 Are all attention modules equally beneficial for SR, or can some hinder performance?
• RQ3 Can a dynamic two-branch attention design improve SR performance efficiently by adjusting branch contributions per input?
• RQ4 How does the proposed A2N compare to state-of-the-art SR models of similar size across common datasets?

Key findings

• Attention modules at different layers respond differently; early layers tend to emphasize low-frequency components while later layers emphasize high-frequency details.
• Uniformly applying attention across all layers yields suboptimal gains; selectively applying attention to certain blocks or employing dynamic weighting can be more effective.
• The A2 structure enables a low-cost, two-branch fusion (attention and non-attention) with dynamic, input-dependent weighting, improving capacity with little parameter overhead.
• A2N achieves competitive or superior PSNR/SSIM compared with state-of-the-art lightweight SR models for scale factors x2, x3, and x4 on standard benchmarks.
• On Manga109 with scale x3, A2N-M outperforms AWSRN-M by 0.15 dB in PSNR; on Urban100 and Manga109 with larger scales, A2N exceeds IMDN by more than 0.2 dB in PSNR.
• Dynamic Attention Module (DAM) shows higher diffusion index (DI) and broader input utilization (LAM analysis) compared to baseline methods, indicating more effective use of LR information.