[Paper Review] Hierarchical Residual Attention Network for Single Image Super-Resolution

Parichehr Behjati, Pau Rodríguez+4 more|arXiv (Cornell University)|Dec 8, 2020

Advanced Image Processing Techniques39 references4 citations

TL;DR

This paper proposes Hierarchical Residual Attention Network (HRAN), a lightweight single image super-resolution model that uses parallel residual and attention paths with hierarchical feature and attention banks to preserve high-frequency details. By decoupling attention computation from residual blocks and aggregating features across all residual blocks, HRAN achieves state-of-the-art performance with minimal computational cost, improving PSNR by up to 0.21 dB over ResNet and 0.3 dB over EDSR on Urban100 at ×4 scale.

ABSTRACT

Convolutional neural networks are the most successful models in single image super-resolution. Deeper networks, residual connections, and attention mechanisms have further improved their performance. However, these strategies often improve the reconstruction performance at the expense of considerably increasing the computational cost. This paper introduces a new lightweight super-resolution model based on an efficient method for residual feature and attention aggregation. In order to make an efficient use of the residual features, these are hierarchically aggregated into feature banks for posterior usage at the network output. In parallel, a lightweight hierarchical attention mechanism extracts the most relevant features from the network into attention banks for improving the final output and preventing the information loss through the successive operations inside the network. Therefore, the processing is split into two independent paths of computation that can be simultaneously carried out, resulting in a highly efficient and effective model for reconstructing fine details on high-resolution images from their low-resolution counterparts. Our proposed architecture surpasses state-of-the-art performance in several datasets, while maintaining relatively low computation and memory footprint.

Motivation & Objective

• To address the degradation of high-frequency features in deep residual networks used for single image super-resolution (SISR).
• To reduce computational and memory costs in SISR models without sacrificing reconstruction quality.
• To prevent information loss from in-place attention mechanisms that discard relevant features during residual aggregation.
• To improve feature utilization across the network hierarchy by introducing dedicated feature and attention banks.
• To develop a lightweight, efficient architecture that outperforms state-of-the-art models in PSNR with comparable parameter counts.

Proposed method

• Introduces Residual Attention Feature Groups (RAFGs) that stack residual blocks and lightweight channel attention (LCA) blocks in parallel paths.
• Uses a hierarchical feature bank to aggregate outputs from all residual blocks, preserving multi-level representations.
• Employs a separate attention bank to extract and preserve the most relevant features through a dedicated computational path.
• Applies 1×1 convolutions every three residual blocks to enable efficient feature fusion and gradient flow.
• Decouples attention computation from residual blocks to avoid loss of high-frequency details during residual addition.
• Designs the network with two independent paths: one for residual feature learning and one for attention-based feature refinement, enabling parallel processing.

Experimental results

Research questions

• RQ1 Can a parallel architecture with dedicated attention paths improve feature preservation in deep SISR networks?
• RQ2 Does decoupling attention from residual blocks reduce information loss during feature propagation?
• RQ3 Can hierarchical feature and attention banks enhance reconstruction quality without increasing model complexity?
• RQ4 How does the proposed RAFG design compare to standard residual stacking in terms of PSNR and computational cost?
• RQ5 To what extent do feature and attention banks contribute to performance gains in SISR models?

Key findings

• HRAN achieves a PSNR of 28.76 dB on Set14 and 26.39 dB on Urban100 at ×4 scale, outperforming EDSR and RCAN with similar parameter counts.
• The proposed hierarchical feature bank improves PSNR by 0.21 dB over a standard ResNet baseline (26.03 dB → 26.24 dB) with only 65K additional parameters.
• Removing the feature and attention banks reduces PSNR by 0.06 dB on average, confirming their positive impact on feature distillation.
• The proposed parallel attention mechanism (LCA) achieves 28.76 dB PSNR on Set14 and 26.39 dB on Urban100, outperforming in-place attention by 0.07 dB.
• The LCA mechanism increases PSNR by 0.07 dB over in-place attention while adding only 66K parameters, demonstrating efficiency and effectiveness.
• HRAN maintains low computational and memory footprint, making it suitable for real-world deployment on resource-constrained devices.