[Paper Review] Intriguing Findings of Frequency Selection for Image Deblurring

Xintian Mao, Yiming Liu+4 more|arXiv (Cornell University)|Nov 23, 2021

Advanced Image Processing TechniquesComputer Science54 references37 citations

TL;DR

The paper reveals that applying ReLU in the frequency domain followed by inverse FFT reveals blur patterns, and introduces a plug-and-play Res FFT-ReLU Block to fuse frequency-domain blur cues with pixel-domain features, boosting deblurring performance across multiple architectures and datasets.

ABSTRACT

Blur was naturally analyzed in the frequency domain, by estimating the latent sharp image and the blur kernel given a blurry image. Recent progress on image deblurring always designs end-to-end architectures and aims at learning the difference between blurry and sharp image pairs from pixel-level, which inevitably overlooks the importance of blur kernels. This paper reveals an intriguing phenomenon that simply applying ReLU operation on the frequency domain of a blur image followed by inverse Fourier transform, i.e., frequency selection, provides faithful information about the blur pattern (e.g., the blur direction and blur level, implicitly shows the kernel pattern). Based on this observation, we attempt to leverage kernel-level information for image deblurring networks by inserting Fourier transform, ReLU operation, and inverse Fourier transform to the standard ResBlock. 1x1 convolution is further added to let the network modulate flexible thresholds for frequency selection. We term our newly built block as Res FFT-ReLU Block, which takes advantages of both kernel-level and pixel-level features via learning frequency-spatial dual-domain representations. Extensive experiments are conducted to acquire a thorough analysis on the insights of the method. Moreover, after plugging the proposed block into NAFNet, we can achieve 33.85 dB in PSNR on GoPro dataset. Our method noticeably improves backbone architectures without introducing many parameters, while maintaining low computational complexity. Code is available at https://github.com/DeepMed-Lab/DeepRFT-AAAI2023.

Motivation & Objective

• Motivate deblurring research to consider blur kernels and frequency-domain cues, not just pixel-level differences.
• Demonstrate that frequency-domain ReLU can reveal blur patterns and kernel information from blurry images.
• Propose a lightweight, plug-and-play FFT-based block to fuse kernel-level and pixel-level features in standard ResBlocks.
• Show that integrating the FFT-ReLU stream improves backbones like MIMO-UNet and NAFNet with minimal parameter overhead.

Proposed method

• Demonstrate that applying ReLU on the Fourier transform of a blurry image and inverse transforming separates blur-pattern components.
• Propose the Res FFT-ReLU Block that adds an FFT-ReLU stream to a standard ResBlock, including 2D real FFT, 1x1 convolutions, ReLU, and inverse 2D real FFT.
• Introduce learnable complex 1x1 convolutions in the frequency domain to modulate flexible frequency thresholds.
• Replace the identity mapping in ResBlock with the FFT-ReLU-based path and fuse with the spatial residual stream.
• Provide ablations to study the position of the FFT-ReLU stream, threshold effects, and complex vs real convolutions.
• Evaluate on GoPro, HIDE, RealBlur datasets and report PSNR/SSIM comparisons, parameters, FLOPs, and inference time.

Experimental results

Research questions

• RQ1 Can frequency-domain processing reveal faithful blur patterns and kernel information from a blurry image?
• RQ2 Does incorporating a frequency-domain FFT-ReLU stream into ResBlocks improve deblurring performance across backbone architectures?
• RQ3 How do threshold choices and complex-valued convolutions in the frequency stream affect deblurring results?
• RQ4 What are the empirical gains when plugging the proposed Res FFT-ReLU Block into existing architectures like MIMO-UNet and NAFNet?

Key findings

• Frequency-domain ReLU followed by inverse FFT can reveal blur-pattern components, aiding kernel-level information extraction.
• Inserting the FFT-ReLU stream into ResBlocks yields substantial PSNR gains across architectures (e.g., RSNet from 28.06 to 30.30 PSNR on GoPro with FFT-ReLU).
• The Res FFT-ReLU Block is plug-and-play and improves multiple backbones (e.g., RSNet, U-Net, MPRNet, MIMO-UNet, NAFNet) on GoPro and HIDE datasets.
• Using learnable complex 1x1 convolutions in the frequency stream and an FFT-ReLU pathway provides better flexibility and performance than real-only or simple concatenation alternatives.
• Replacing the identity mapping in ResBlock with the FFT-ReLU stream enables learning global context through frequency-domain nonlinearities, enhancing deblurring performance.
• FMIMO-UNet and FNAFNet variants incorporating Res FFT-ReLU achieve state-of-the-art or competitive PSNR scores on GoPro, HIDE, and RealBlur datasets (e.g., FMIMO-UNet+ PSNRs: 33.52 on GoPro, 31.66 on HIDE, 36.11 RealBlur-R, 28.88 RealBlur-J; FNAFNet64 PSNR on GoPro: 33.85).