[Paper Review] Restormer: Efficient Transformer for High-Resolution Image Restoration

Syed Waqas Zamir, Aditya Arora+4 more|arXiv (Cornell University)|Nov 18, 2021

Advanced Image Processing Techniques99 references191 citations

TL;DR

Restormer introduces a lightweight Transformer with multi-Dconv head transposed attention and gated-Dconv feed-forward networks to enable high-resolution image restoration with linear complexity, achieving state-of-the-art results across multiple tasks.

ABSTRACT

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.

Motivation & Objective

• Motivate image restoration as an ill-posed problem requiring strong image priors and long-range dependencies.
• Overcome the quadratic complexity of standard self-attention to enable high-resolution restoration.
• Propose Restormer with novel building blocks (MDTA and GDFN) and a progressive training strategy for multi-scale context learning.

Proposed method

• Introduce an encoder-decoder architecture that processes high-resolution images without partitioning into local windows.
• Replace vanilla multi-head self-attention with multi-Dconv head transposed attention (MDTA) that computes cross-channel covariance with linear complexity, incorporating local context via 1x1 and depthwise convolutions.
• Propose gated-Dconv feed-forward network (GDFN) that uses a gating mechanism and depthwise convolutions to control and enrich feature transformations.
• Use a progressive learning strategy that starts training on small patches with large batches and gradually shifts to larger patches with smaller batches to capture global image statistics.
• Train task-specific Restormer models for deraining, deblurring, defocus deblurring (single-image and dual-pixel), and denoising, with an adversarially small number of parameters and FLOPs.

Experimental results

Research questions

• RQ1 Can Restormer model global pixel interactions with linear complexity suitable for high-resolution image restoration?
• RQ2 How do the proposed MDTA and GDFN components compare to traditional attention and feed-forward networks in restoration tasks?
• RQ3 Does progressive learning improve performance on full-resolution images across multiple restoration tasks?
• RQ4 What are the state-of-the-art performances of Restormer on deraining, motion deblurring, defocus deblurring, and denoising datasets?

Key findings

• Restormer achieves state-of-the-art results on image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel), and image denoising across multiple datasets.
• On average, Restormer outperforms the previous best deraining method by 1.05 dB (across five Rain datasets).
• For motion deblurring, Restormer improves averaged PSNR/SSIM by 0.47 dB over MIMO-UNet+ and 0.26 dB over MPRNet, while having 81% fewer FLOPs than MPRNet and 4.4x fewer parameters than IPT, with 29x faster runtime.
• In Gaussian grayscale/color denoising and real image denoising, Restormer matches or exceeds leading CNN/Transformer methods and attains higher PSNR in real-image denoising on SIDD/DND benchmarks.
• Restormer demonstrates strong generalization, being trained on GoPro for deblurring yet achieving state-of-the-art performance on other datasets.
• Ablation studies show that the combination of MDTA with GDFN yields the best PSNR across high-resolution urban datasets, validating the design choices.