[Paper Review] Unfolding the Alternating Optimization for Blind Super Resolution

Zhengxiong Luo, Yan Huang+3 more|arXiv (Cornell University)|Oct 6, 2020

Advanced Image Processing Techniques38 references146 citations

TL;DR

Proposes an end-to-end deep alternating network (DAN) that jointly estimates blur kernels and restores high-resolution images in blind SR by unfolding an alternating optimization into a trainable network. It outperforms state-of-the-art methods with higher speed and better visual results.

ABSTRACT

Previous methods decompose blind super resolution (SR) problem into two sequential steps: extit{i}) estimating blur kernel from given low-resolution (LR) image and extit{ii}) restoring SR image based on estimated kernel. This two-step solution involves two independently trained models, which may not be well compatible with each other. Small estimation error of the first step could cause severe performance drop of the second one. While on the other hand, the first step can only utilize limited information from LR image, which makes it difficult to predict highly accurate blur kernel. Towards these issues, instead of considering these two steps separately, we adopt an alternating optimization algorithm, which can estimate blur kernel and restore SR image in a single model. Specifically, we design two convolutional neural modules, namely extit{Restorer} and extit{Estimator}. extit{Restorer} restores SR image based on predicted kernel, and extit{Estimator} estimates blur kernel with the help of restored SR image. We alternate these two modules repeatedly and unfold this process to form an end-to-end trainable network. In this way, extit{Estimator} utilizes information from both LR and SR images, which makes the estimation of blur kernel easier. More importantly, extit{Restorer} is trained with the kernel estimated by extit{Estimator}, instead of ground-truth kernel, thus extit{Restorer} could be more tolerant to the estimation error of extit{Estimator}. Extensive experiments on synthetic datasets and real-world images show that our model can largely outperform state-of-the-art methods and produce more visually favorable results at much higher speed. The source code is available at https://github.com/greatlog/DAN.git.

Motivation & Objective

• Motivate blind SR where blur kernels are unknown and single-step kernel estimation and SR are poorly matched.
• Propose a unified, end-to-end network that alternates between kernel estimation and image restoration.
• Ensure robustness to kernel estimation errors by training restoration with estimated kernels.
• Improve training and inference speed compared with two-step blind SR pipelines.
• Demonstrate superior performance on synthetic and real-world images across multiple datasets.

Proposed method

• Introduce two convolutional modules, Estimator and Restorer, to implement an alternating optimization for kernel estimation and SR.
• Unfold the iteration into a fixed-depth deep network (DAN) that is end-to-end trainable.
• Use a conditional residual block (CRB) to couple basic inputs with conditional inputs (SR image and kernel), ensuring outputs stay aligned with their conditioning inputs.
• Estimator takes both LR and SR images to predict a PCA-reduced blur kernel; Restorer takes LR image and the estimated kernel to recover the SR image.
• Train Restorer with the Estimator's output kernel (not ground-truth), improving tolerance to estimation errors.
• Evaluate with two degradation settings (Gaussian blur and irregular kernels), showing robustness and speed advantages.

Experimental results

Research questions

• RQ1 Can an end-to-end network based on alternating optimization outperform traditional two-step blind SR pipelines?
• RQ2 Does sharing and iterating Estimator and Restorer improve kernel estimation and SR quality when both are trained jointly?
• RQ3 How does DAN perform on synthetic and real-world images in terms of PSNR/SSIM and inference speed?
• RQ4 Is the network robust to kernel estimation errors and varying iteration counts during inference?

Key findings

• DAN substantially outperforms state-of-the-art blind SR methods on synthetic datasets across multiple scales and datasets.
• DAN achieves dramatically faster inference (0.75s per image on RTX2080Ti) than KernelGAN+ZSSR and outperforms IKC in PSNR/SSIM on several benchmarks.
• Jointly training Estimator and Restorer with alternating optimization yields better kernel estimation (lower L1 error in reduced kernel space) and improved SR results, especially as kernel variance increases.
• Allowing more iterations during testing can further improve or stabilize performance, indicating learned cooperation between modules beyond a fixed point.
• Experiments on real-world noisy images show DAN can denoise and restore details more reliably than baselines.