[Paper Review] Image Super-Resolution Using Very Deep Residual Channel Attention Networks

Yulun Zhang, Kunpeng Li+4 more|arXiv (Cornell University)|Jul 8, 2018

Advanced Image Processing TechniquesComputer Science40 references264 citations

TL;DR

RCAN introduces a very deep residual channel attention network with residual-in-residual (RIR) architecture and channel attention (CA) to achieve state-of-the-art image super-resolution, especially at high scales.

ABSTRACT

Convolutional neural network (CNN) depth is of crucial importance for image super-resolution (SR). However, we observe that deeper networks for image SR are more difficult to train. The low-resolution inputs and features contain abundant low-frequency information, which is treated equally across channels, hence hindering the representational ability of CNNs. To solve these problems, we propose the very deep residual channel attention networks (RCAN). Specifically, we propose a residual in residual (RIR) structure to form very deep network, which consists of several residual groups with long skip connections. Each residual group contains some residual blocks with short skip connections. Meanwhile, RIR allows abundant low-frequency information to be bypassed through multiple skip connections, making the main network focus on learning high-frequency information. Furthermore, we propose a channel attention mechanism to adaptively rescale channel-wise features by considering interdependencies among channels. Extensive experiments show that our RCAN achieves better accuracy and visual improvements against state-of-the-art methods.

Motivation & Objective

• Motivate deeper networks for image SR and overcome training difficulty.
• Improve learning by bypassing abundant low-frequency information through skip connections.
• Introduce channel attention to adaptively emphasize informative feature channels.
• Demonstrate that very deep RCAN outperforms previous SR methods across standard benchmarks.

Proposed method

• Propose residual in residual (RIR) structure with long skip connections to enable very deep networks (>400 layers).
• Use residual groups (RG) containing multiple RCABs with short skip connections to build depth.
• Incorporate channel attention (CA) to model interdependencies among feature channels and rescale channel-wise features.
• Embed RCABs (residual channel attention blocks) inside RGs to learn residual high-frequency information.
• Apply an upscale module (ESPCNN-based) at the end to produce high-resolution outputs.
• Train with L1 loss on LR→HR mappings from DIV2K and evaluate on standard SR benchmarks.

Experimental results

Research questions

• RQ1 Can extremely deep networks improve SR performance beyond existing deep SR models?
• RQ2 Does residual in residual (RIR) structure stabilize training of very deep SR networks?
• RQ3 Does channel attention (CA) effectively guide learning toward high-frequency channel-wise features in SR?
• RQ4 How does RCAN compare to state-of-the-art SR methods across multiple scaling factors and datasets?

Key findings

• RCAN achieves superior PSNR/SSIM on standard benchmarks (Set5, Set14, B100, Urban100, Manga109) across 2×, 3×, 4×, and 8× scales compared to prior methods.
• Self-ensembling (RCAN+) further improves results over RCAN without increasing depth; RCAN+ consistently yields best results.
• RIR significantly stabilizes training and enables networks with over 400 convolutional layers to deliver substantial SR gains.
• Channel attention (CA) provides additional gains by adaptively rescaling channel-wise features, especially when used with very deep architectures.
• RCAN achieves notable improvements on challenging datasets like Urban100 and Manga109 at large scales, outperforming deeper but less flexible baselines like EDSR.