[Paper Review] SwinFIR: Revisiting the SwinIR with Fast Fourier Convolution and Improved Training for Image Super-Resolution

Dafeng Zhang, Feiyu Huang+3 more|arXiv (Cornell University)|Aug 24, 2022

Advanced Optical Sensing Technologies72 citations

TL;DR

SwinFIR extends SwinIR by introducing a Spatial Frequency Block based on Fast Fourier Convolution to capture global information, along with data augmentation and a feature ensemble, achieving state-of-the-art results in image super-resolution.

ABSTRACT

Transformer-based methods have achieved impressive image restoration performance due to their capacities to model long-range dependency compared to CNN-based methods. However, advances like SwinIR adopts the window-based and local attention strategy to balance the performance and computational overhead, which restricts employing large receptive fields to capture global information and establish long dependencies in the early layers. To further improve the efficiency of capturing global information, in this work, we propose SwinFIR to extend SwinIR by replacing Fast Fourier Convolution (FFC) components, which have the image-wide receptive field. We also revisit other advanced techniques, i.e, data augmentation, pre-training, and feature ensemble to improve the effect of image reconstruction. And our feature ensemble method enables the performance of the model to be considerably enhanced without increasing the training and testing time. We applied our algorithm on multiple popular large-scale benchmarks and achieved state-of-the-art performance comparing to the existing methods. For example, our SwinFIR achieves the PSNR of 32.83 dB on Manga109 dataset, which is 0.8 dB higher than the state-of-the-art SwinIR method.

Motivation & Objective

• Motivate improving image super-resolution by leveraging global information in early layers.
• Design a global-feature extractor that integrates Fast Fourier Convolution with local CNN features.
• Explore training-time enhancements (loss, data augmentation, pre-training) to boost SR performance.
• Propose a zero-cost post-processing ensemble to boost results without extra training or inference time.

Proposed method

• Replace the SwinIR convolution in deep feature extraction with the Spatial Frequency Block (SFB) that combines a spatial CNN path and a frequency-domain FFT path.
• Use Fast Fourier Convolution (FFC) in the frequency branch to capture global information and fuse with a residual spatial path.
• Adopt Charbonnier loss instead of L1/L2 for training stability and performance.
• Apply pixel-domain data augmentations (channel shuffle, Mixup, etc.) to improve generalization.
• Introduce a feature ensemble post-processing technique that ensembles multiple trained models without increasing training/testing time.

Experimental results

Research questions

• RQ1 Can the Spatial Frequency Block effectively capture global information in SR and outperform purely local-window transformers?
• RQ2 Do data augmentation strategies like channel shuffle and Mixup significantly improve SR performance in this setting?
• RQ3 Does a zero-cost feature ensemble reliably boost SR results across datasets without extra compute?
• RQ4 How does SwinFIR compare to SwinIR and EDT on standard SR benchmarks when using pre-training and larger windows?

Key findings

• SwinFIR achieves PSNR of 32.83 dB on Manga109, which is 0.80 dB higher than the state-of-the-art SwinIR method.
• Across Manga109 and Urban100, SwinFIR provides improvements in PSNR of about 0.30 to 0.80 dB over SwinIR and 0.24 to 0.44 dB over EDT.
• SwinFIR-T (lightweight variant) attains 31.50 dB PSNR on Manga109 with comparable parameter counts to SwinIR/EDT-T, surpassing them by 0.58 dB and 0.15 dB respectively.
• The proposed data augmentations (e.g., channel shuffle, Mixup) and feature ensemble post-processing yield notable gains without increasing training/testing time.
• SwinFIR and its stereo SR variant SwinFIRSSR demonstrate strong performance across classical, lightweight, and stereo SR benchmarks, including KITTI, Middlebury, and Flickr1024 datasets.