[Paper Review] Residual Non-local Attention Networks for Image Restoration

Yulun Zhang, Kunpeng Li+3 more|arXiv (Cornell University)|Mar 24, 2019

Image and Signal Denoising Methods203 citations

TL;DR

RNAN introduces residual local and non-local attention blocks with trunk and mask branches, enabling non-local mixed attention and residual attention learning to boost image restoration across denoising, demosaicing, compression artifacts reduction, and super-resolution.

ABSTRACT

In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention block. The trunk branch is used to extract hierarchical features. Local and non-local mask branches aim to adaptively rescale these hierarchical features with mixed attentions. The local mask branch concentrates on more local structures with convolutional operations, while non-local attention considers more about long-range dependencies in the whole feature map. Furthermore, we propose residual local and non-local attention learning to train the very deep network, which further enhance the representation ability of the network. Our proposed method can be generalized for various image restoration applications, such as image denoising, demosaicing, compression artifacts reduction, and super-resolution. Experiments demonstrate that our method obtains comparable or better results compared with recently leading methods quantitatively and visually.

Motivation & Objective

• Motivate improvement over local CNNs by capturing long-range dependencies in corrupted images.
• Develop very deep networks with residual local and non-local attention to enhance restoration quality.
• Integrate trunk and mask branches to learn hierarchical features and adaptively rescale them with mixed attention.
• Incorporate non-local attention to exploit global dependencies while preserving low-level details.
• Demonstrate generalization of RNAN across multiple image restoration tasks.

Proposed method

• Design trunk branches that extract hierarchical features via residual blocks.
• Implement mask branches that generate attention maps through local convolutions and large receptive fields via downsampling/upsampling or non-local blocks.
• Incorporate non-local block in the mask branch to produce non-local mixed attention.
• Propose residual non-local attention learning H_RNA(x)=H_trunk(x)H_mask(x)+x to preserve low-level details in restoration.
• Train RNAN with L2 loss on low-/high-quality image pairs and use very deep stacking of RNABs (local and non-local) to enable strong representation.
• Provide implementation specifics: 10 RNAN blocks, 64/32 feature channels, ADAM optimizer, 1x1 and 3x3 kernels, etc.

Experimental results

Research questions

• RQ1 Can residual local and non-local attention improve restoration quality over standard CNN-based approaches?
• RQ2 Does incorporating non-local mixed attention in the mask branch enhance the network’s ability to handle long-range dependencies in degraded images?
• RQ3 Is the proposed residual non-local attention learning effective for very deep networks in image restoration across tasks like denoising, demosaicing, compression artifact reduction, and super-resolution?
• RQ4 What is the impact of mask branch design and non-local blocks on performance and efficiency?

Key findings

• RNAN achieves superior results compared to leading methods across denoising, demosaicing, compression artifacts reduction, and SR in the reported experiments.
• Non-local mixed attention and the mask branch consistently improve restoration performance over configurations without them.
• More non-local blocks yield better PSNR, but with higher computational cost, motivating a choice of two non-local blocks for a balance.
• RNAN provides state-of-the-art results with moderate model size (~7.4M parameters) and competitive runtimes on standard benchmarks.
• Self-ensemble RNAN+ further improves SR results, achieving strong performance while remaining more parameter-efficient than some deeper SR models.