[Paper Review] Efficient Mixed Transformer for Single Image Super-Resolution

Ling Zheng, Jinchen Zhu+2 more|arXiv (Cornell University)|May 19, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes Efficient Mixed Transformer (EMT), a lightweight Transformer-based architecture for single image super-resolution (SISR) that enhances local feature aggregation via a parameter-free Pixel Mixer (PM) and improves global modeling efficiency using striped window self-attention (SWSA). EMT achieves state-of-the-art performance with reduced complexity, outperforming existing methods on benchmark datasets like Set5, Urban100, and BSD100 at ×4 upscaling.

ABSTRACT

Recently, Transformer-based methods have achieved impressive results in single image super-resolution (SISR). However, the lack of locality mechanism and high complexity limit their application in the field of super-resolution (SR). To solve these problems, we propose a new method, Efficient Mixed Transformer (EMT) in this study. Specifically, we propose the Mixed Transformer Block (MTB), consisting of multiple consecutive transformer layers, in some of which the Pixel Mixer (PM) is used to replace the Self-Attention (SA). PM can enhance the local knowledge aggregation with pixel shifting operations. At the same time, no additional complexity is introduced as PM has no parameters and floating-point operations. Moreover, we employ striped window for SA (SWSA) to gain an efficient global dependency modelling by utilizing image anisotropy. Experimental results show that EMT outperforms the existing methods on benchmark dataset and achieved state-of-the-art performance. The Code is available at https://github.com/Fried-Rice-Lab/FriedRiceLab.

Motivation & Objective

• To address the lack of locality modeling and high computational complexity in Transformer-based SISR methods.
• To improve local feature aggregation without increasing parameters or FLOPs.
• To enhance global dependency modeling through anisotropic image structure exploitation.
• To achieve a better trade-off between performance and model complexity for lightweight SISR.
• To enable efficient deployment of SISR models on mobile and embedded devices.

Proposed method

• Proposes the Mixed Transformer Block (MTB) with alternating global (GTL) and local (LTL) transformer layers to balance global and local modeling.
• Introduces the Pixel Mixer (PM) as a parameter-free, FLOP-free local perceptron using channel-wise separation and pixel shifting to enhance local feature aggregation.
• Employs Striped Window Self-Attention (SWSA) that leverages image anisotropy to efficiently model long-range dependencies with reduced computation.
• Uses a three-stage architecture: shallow feature extraction, deep feature extraction via stacked MTBs, and reconstruction with skip connections.
• Applies channel-wise feature mixing and pixel shifting in PM to expand local receptive fields without additional parameters.
• Controls model complexity by replacing selected self-attention layers with PM while maintaining performance.

Experimental results

Research questions

• RQ1 Can a mixed Transformer block with selective self-attention and local perceptrons improve local feature learning in SISR?
• RQ2 Does a parameter-free Pixel Mixer enhance local knowledge aggregation without increasing FLOPs or parameters?
• RQ3 Can striped window attention improve global dependency modeling efficiency by exploiting image anisotropy?
• RQ4 What is the optimal balance between global and local layers in the MTB for SISR performance and complexity?
• RQ5 Can the proposed EMT achieve state-of-the-art performance with lower model complexity than existing lightweight SISR methods?

Key findings

• EMT achieves SOTA performance on Set5, Set14, Urban100, BSD100, and Manga109 at ×4 super-resolution, with PSNR of 32.559 on Set5 and 26.741 on Urban100.
• The MTB with 2GTL (global transformer layers) achieves the best performance-complexity trade-off, outperforming 4GTL and 6GTL variants at equal parameter levels.
• The Pixel Mixer (PM) reduces Mean Attention Distance (MAD) and increases local attention usage, especially in deeper layers, without adding parameters or FLOPs.
• EMT with PM achieves 31.329 PSNR on Manga109, outperforming the identity mapping baseline (31.296) under identical parameter constraints.
• SWSA with striped windows improves global modeling efficiency and performance compared to square windows, especially in deeper network stages.
• The ablation study confirms that PM significantly improves texture and detail recovery, as shown in qualitative results on Urban100 and Manga109.