[Paper Review] Transformer for Single Image Super-Resolution

Zhisheng Lu, Juncheng Li+4 more|arXiv (Cornell University)|Aug 25, 2021

Optical Coherence Tomography Applications5 citations

TL;DR

This paper proposes Efficient Super-Resolution Transformer (ESRT), a hybrid CNN-Transformer architecture that achieves state-of-the-art performance in single image super-resolution with significantly reduced computational cost. By integrating a Lightweight CNN Backbone (LCB) with High Preserving Blocks and a Lightweight Transformer Backbone (LTB) using Efficient Multi-Head Attention (EMHA), ESRT reduces GPU memory usage to 4,191M—only 26% of the original Transformer—while improving PSNR by up to 0.32dB on benchmark datasets.

ABSTRACT

Single image super-resolution (SISR) has witnessed great strides with the development of deep learning. However, most existing studies focus on building more complex networks with a massive number of layers. Recently, more and more researchers start to explore the application of Transformer in computer vision tasks. However, the heavy computational cost and high GPU memory occupation of the vision Transformer cannot be ignored. In this paper, we propose a novel Efficient Super-Resolution Transformer (ESRT) for SISR. ESRT is a hybrid model, which consists of a Lightweight CNN Backbone (LCB) and a Lightweight Transformer Backbone (LTB). Among them, LCB can dynamically adjust the size of the feature map to extract deep features with a low computational cost. LTB is composed of a series of Efficient Transformers (ET), which occupies a small GPU memory occupation, thanks to the specially designed Efficient Multi-Head Attention (EMHA). Extensive experiments show that ESRT achieves competitive results with low computational costs. Compared with the original Transformer which occupies 16,057M GPU memory, ESRT only occupies 4,191M GPU memory. All codes are available at https://github.com/luissen/ESRT.

Motivation & Objective

• To address the high computational cost and GPU memory consumption of Vision Transformers in single image super-resolution (SISR).
• To enhance long-range feature dependencies between similar image patches using a lightweight Transformer mechanism.
• To develop a hybrid CNN-Transformer model that maintains high performance while drastically reducing model size and inference cost.
• To enable practical deployment of Transformer-based SISR models in real-world applications with limited hardware resources.

Proposed method

• Proposes a Lightweight CNN Backbone (LCB) that dynamically adjusts feature map size to reduce computation while preserving high-frequency details via a High-frequency Filtering Module (HFM).
• Introduces an Adaptive Residual Feature Block (ARFB) that adaptively fuses residual and identity paths to improve feature representation.
• Designs an Efficient Transformer (ET) with a novel Efficient Multi-Head Attention (EMHA) mechanism that limits self-attention to local regions, reducing GPU memory usage.
• Employs a splitting factor in EMHA to compute attention over localized image blocks, enabling efficient modeling of long-range dependencies without full-attention computation.
• Combines LCB and LTB in a hybrid architecture, where CNN extracts deep features and Transformer models long-range context across similar patches.
• Trains the model end-to-end on standard SISR benchmarks (Set5, Set14, Urban100) with a patch size of 48×48 and batch size of 16.

Figure 2 : The architecture of the proposed Efficient Super-Resolution Transformer. Among them, LCB, LTB, HPB, and ET stand for the Lightweight CNN Backbone, the Lightweight Transformer Backbone, high preserving block, and efficient Transformers, respectively.

Experimental results

Research questions

• RQ1 Can a lightweight Transformer mechanism effectively model long-range dependencies between similar image patches in SISR without excessive memory cost?
• RQ2 How can a hybrid CNN-Transformer architecture improve super-resolution performance while minimizing computational and memory overhead?
• RQ3 To what extent can the proposed Efficient Multi-Head Attention (EMHA) reduce GPU memory usage compared to standard Transformers in SISR?
• RQ4 Can the proposed Efficient Transformer (ET) be universally applied to improve existing SISR models like RCAN with minimal architectural changes?
• RQ5 Does the ESRT model achieve better performance on real-world images compared to dedicated lightweight models such as IMDN and LK-KPN?

Key findings

• ESRT achieves a PSNR of 32.18 dB on Set5 at ×4 super-resolution, outperforming the original Transformer (32.14 dB) with 74% less GPU memory usage (4,191M vs. 16,057M).
• The proposed Efficient Transformer (ET) reduces GPU memory consumption to 1/4 of the original Transformer while improving performance by 0.04 dB on Set5.
• When integrated into RCAN, the ET module (RCAN/2+ET) achieves comparable or better performance than the full RCAN model with 46% fewer parameters (8.7M vs. 16M).
• On real-world images (RealSR dataset), ESRT achieves a PSNR of 28.78 dB at ×4, outperforming IMDN (28.68 dB) and LK-KPN (28.65 dB).
• Compared to SwinIR, ESRT achieves nearly identical performance (29.15 dB on BSD100) with 12% fewer parameters (770K vs. 886K) and 39% less GPU memory (4,191M vs. 6,966M).
• The ablation study confirms that the combination of HFM and ARFB in the LCB improves feature preservation and reduces computation, while EMHA in LTB enables efficient modeling of long-range dependencies.

Figure 3 : The architecture of the proposed High Preserving Block (HPB), which mainly consists of High-frequency Filtering Module (HFM) and Adaptive Residual Feature Blocks (ARFBs).