[Paper Review] SwinIR: Image Restoration Using Swin Transformer

Jingyun Liang, Jiezhang Cao+4 more|arXiv (Cornell University)|Aug 23, 2021

Advanced Image Processing TechniquesComputer Science90 references80 citations

TL;DR

SwinIR uses a Swin Transformer–based architecture with shallow/deep feature extraction and residual Swin Transformer blocks to achieve competitive image restoration across SR, denoising, and JPEG artifact reduction with fewer parameters.

ABSTRACT

Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image reconstruction. In particular, the deep feature extraction module is composed of several residual Swin Transformer blocks (RSTB), each of which has several Swin Transformer layers together with a residual connection. We conduct experiments on three representative tasks: image super-resolution (including classical, lightweight and real-world image super-resolution), image denoising (including grayscale and color image denoising) and JPEG compression artifact reduction. Experimental results demonstrate that SwinIR outperforms state-of-the-art methods on different tasks by $ extbf{up to 0.14$\sim$0.45dB}$, while the total number of parameters can be reduced by $ extbf{up to 67%}$.

Motivation & Objective

• Motivate and demonstrate the effectiveness of Transformer-based models for image restoration.
• Propose SwinIR, a Swin Transformer–based architecture for high-quality image restoration tasks.
• Show that SwinIR can outperform state-of-the-art CNN-based methods with fewer parameters.

Proposed method

• Propose three-module SwinIR: shallow feature extraction, deep feature extraction, and high-quality image reconstruction.
• Deep features are extracted via K residual Swin Transformer blocks, each containing several Swin Transformer layers with a residual convolutional path.
• Fuse shallow and deep features in a reconstruction module to produce the high-quality image, with a skip connection to preserve low-frequency information.
• Use a 3x3 convolution after deep feature extraction to introduce inductive bias before feature fusion.
• Optimize with L1 loss for classical SR and real-world SR, with Charbonnier loss for denoising and JPEG artifact reduction, and optionally GAN/perceptual losses for real-world SR.

Experimental results

Research questions

• RQ1 Can Swin Transformer–based architectures outperform CNN-based methods on classical, real-world, and lightweight image SR?
• RQ2 Are residual Swin Transformer blocks effective for restoring high-frequency details while maintaining model efficiency?
• RQ3 How does SwinIR perform on image denoising and JPEG artifact reduction compared to state-of-the-art methods?
• RQ4 What is the impact of architectural choices such as residual connections and convolutional end blocks on restoration performance?

Key findings

• SwinIR achieves state-of-the-art or competitive PSNR/SSIM on multiple SR datasets with fewer parameters than many CNN-based methods.
• SwinIR demonstrates strong performance for real-world SR when trained with larger degradation models and datasets, surpassing IPT under certain conditions.
• In ablations, residual connections in RSTB significantly improve PSNR, and 3x3 convolutions outperform 1x1 or multiple small convs for feature enhancement.
• The Swin Transformer–based approach shows good convergence and data efficiency, performing well with DIV2K and DIV2K+Flickr2K training data.
• For denoising, SwinIR outperforms traditional and several CNN-based methods across multiple datasets and noise levels, with fewer parameters than some baselines.