[Paper Review] Blind Super-Resolution Kernel Estimation using an Internal-GAN

Sefi Bell-Kligler, Assaf Shocher+1 more|arXiv (Cornell University)|Sep 14, 2019

Advanced Image Processing Techniques215 citations

TL;DR

KernelGAN trains an image-specific internal GAN on a single LR image to estimate its true SR kernel, enabling true Blind-SR by plugging the kernel into existing SR methods. It achieves state-of-the-art results on real and non-ideal LR images without external data.

ABSTRACT

Super resolution (SR) methods typically assume that the low-resolution (LR) image was downscaled from the unknown high-resolution (HR) image by a fixed 'ideal' downscaling kernel (e.g. Bicubic downscaling). However, this is rarely the case in real LR images, in contrast to synthetically generated SR datasets. When the assumed downscaling kernel deviates from the true one, the performance of SR methods significantly deteriorates. This gave rise to Blind-SR - namely, SR when the downscaling kernel ("SR-kernel") is unknown. It was further shown that the true SR-kernel is the one that maximizes the recurrence of patches across scales of the LR image. In this paper we show how this powerful cross-scale recurrence property can be realized using Deep Internal Learning. We introduce "KernelGAN", an image-specific Internal-GAN, which trains solely on the LR test image at test time, and learns its internal distribution of patches. Its Generator is trained to produce a downscaled version of the LR test image, such that its Discriminator cannot distinguish between the patch distribution of the downscaled image, and the patch distribution of the original LR image. The Generator, once trained, constitutes the downscaling operation with the correct image-specific SR-kernel. KernelGAN is fully unsupervised, requires no training data other than the input image itself, and leads to state-of-the-art results in Blind-SR when plugged into existing SR algorithms.

Motivation & Objective

• Motivate the need for real-world Blind-SR where the SR kernel is unknown and potentially non-ideal.
• Propose an unsupervised, image-specific kernel estimation method that leverages cross-scale patch recurrence.
• Demonstrate that the estimated SR kernel improves SR performance when combined with existing SR algorithms.

Proposed method

• Introduce KernelGAN: an image-specific Internal-GAN with a deep linear generator and a patch-based discriminator trained on crops from the LR image.
• The generator learns a downscaling operation that makes the downscaled image’s patch distribution indistinguishable from the original LR image’s patch distribution.
• Extract an explicit SR-kernel by convolving the generator’s layers; apply a differentiable regularization to enforce kernel properties (sum-to-one, centrality, sparsity).
• Use a deep linear generator (5 layers) to match the SR-kernel while preserving interpretability and optimization stability.
• Derive SR-kernels for multiple scales (e.g., k2 and analytically obtain k4 from k2) to enable multi-scale Blind-SR.
• Train the GAN for about 3,000 iterations with ADAM, and rely on a patch-based D-map to guide G toward the target patch distribution.
• Embed the learned SR-kernel into non-blind SR algorithms to evaluate kernel-estimation accuracy and SR performance.

Experimental results

Research questions

• RQ1 Can an image-specific, unsupervised GAN be used to estimate the true SR-kernel from a single LR image?
• RQ2 Does plugging the image-specific SR-kernel into existing SR methods yield state-of-the-art Blind-SR performance on real/non-ideal LR data?
• RQ3 Is a deep linear generator preferable to a single-layer generator for stable convergence to the image-specific SR-kernel?
• RQ4 How well can kernels for different scales be derived from a single run of KernelGAN?
• RQ5 What are the quantitative gains in PSNR/SSIM when using KernelGAN-estimated kernels compared to prior kernel-estimation methods?

Key findings

Method	×2 PSNR/SSIM	×4 PSNR/SSIM
Bicubic Interpolation	28.731 / 0.8040	25.330 / 0.6795
SotA SR algorithms (with bicubic kernel + ZSSR)	29.102 / 0.8215	25.605 / 0.6911
SotA SR algorithms (with EDSRplus)	29.172 / 0.8216	25.638 / 0.6928
SotA SR algorithms (with RCANplus)	29.198 / 0.8223	25.659 / 0.6936
PDN (NTIRE’18 winner)	-	26.340 / 0.7190
KernelGAN (Ours) + SRMD (NTIRE winner)	29.565 / 0.8564	25.711 / 0.7265
KernelGAN (Ours) + ZSSR	30.363 / 0.8669	26.810 / 0.7316
Ground-truth kernel + SRMD	31.962 / 0.8955	27.375 / 0.7655
Ground-truth kernel + ZSSR (Upper bound)	32.436 / 0.8992	27.527 / 0.7446
Ground-truth kernel baseline (alternative)	-	-

• KernelGAN with ZSSR and other SR methods yields substantial improvements over non-blind SR when kernels are non-ideal or unknown.
• KernelGAN-based kernel estimation outperforms Michaeli & Irani (prior kernel estimator) in both visual and quantitative metrics when used with SR algorithms.
• On a DIV2KRK Blind-SR benchmark, KernelGAN achieves ~1 dB PSNR improvement over prior methods for scale ×2 and ~0.47 dB for ×4 when combined with ZSSR.
• The deep linear generator with a 13×13 receptive field significantly outperforms a single-layer generator for kernel estimation.
• The approach enables an analytical path to obtain higher-scale kernels (e.g., k4) from a trained k2 by kernel convolution/dilation relations.
• Real-image experiments show KernelGAN provides more faithful SR-kernel estimates, leading to higher perceptual and numerical SR quality.