[Paper Review] End-to-end Alternating Optimization for Blind Super Resolution

Zhengxiong Luo, Yan Huang+3 more|arXiv (Cornell University)|May 14, 2021

Advanced Image Processing Techniques50 references30 citations

TL;DR

The paper proposes a deep alternating network (DAN) that jointly estimates the blur kernel and restores a blind super-resolution image in an end-to-end, iteratively unfolded framework, improving compatibility between estimation and restoration and achieving state-of-the-art results with high speed.

ABSTRACT

Previous methods decompose the blind super-resolution (SR) problem into two sequential steps: extit{i}) estimating the blur kernel from given low-resolution (LR) image and extit{ii}) restoring the SR image based on the estimated kernel. This two-step solution involves two independently trained models, which may not be well compatible with each other. A small estimation error of the first step could cause a severe performance drop of the second one. While on the other hand, the first step can only utilize limited information from the LR image, which makes it difficult to predict a highly accurate blur kernel. Towards these issues, instead of considering these two steps separately, we adopt an alternating optimization algorithm, which can estimate the blur kernel and restore the SR image in a single model. Specifically, we design two convolutional neural modules, namely extit{Restorer} and extit{Estimator}. extit{Restorer} restores the SR image based on the predicted kernel, and extit{Estimator} estimates the blur kernel with the help of the restored SR image. We alternate these two modules repeatedly and unfold this process to form an end-to-end trainable network. In this way, extit{Estimator} utilizes information from both LR and SR images, which makes the estimation of the blur kernel easier. More importantly, extit{Restorer} is trained with the kernel estimated by extit{Estimator}, instead of the ground-truth kernel, thus extit{Restorer} could be more tolerant to the estimation error of extit{Estimator}. Extensive experiments on synthetic datasets and real-world images show that our model can largely outperform state-of-the-art methods and produce more visually favorable results at a much higher speed. The source code is available at \url{https://github.com/greatlog/DAN.git}.

Motivation & Objective

• Motivate blind SR where the blur kernel is unknown and varies per image.
• Propose an end-to-end architecture that alternates kernel estimation and SR restoration within a single model.
• Improve compatibility between kernel estimation and SR restoration to reduce error propagation.
• Demonstrate superior quantitative and qualitative performance on synthetic and real-world images while increasing speed.

Proposed method

• Introduce two CNN modules: Estimator (kernel estimator) and Restorer (SR reconstructor).
• Formulate blind SR as an alternating optimization problem and unfold it into a trainable network (DAN) with shared parameters across iterations.
• Use a dual-path conditional block (DPCB) and dual-path conditional group (DPCG) to efficiently fuse basic and conditional inputs without heavy concatenation.
• Predict complete blur kernels with a Softmax to enforce kernel sum-to-one constraint.
• Train end-to-end with last-iteration supervision while keeping intermediate results unconstrainted to encourage convergence.
• Adopt 4 fixed alternating iterations in practice and initialize the kernel with a Dirac delta, reshaped and PCA-reduced for input to the model.

Experimental results

Research questions

• RQ1 Can an end-to-end network jointly estimate blur kernels and perform blind SR more effectively than two-step methods?
• RQ2 Does information from both LR and SR images help Estimator predict better kernels, and does Restorer tolerate Estimator errors when trained jointly?
• RQ3 Do architectural innovations like the Dual-Path Conditional Block improve performance and efficiency in Estimator and Restorer?
• RQ4 What is the impact of supervising Estimator on complete kernels vs reduced kernels on final SR quality?
• RQ5 How do DAN variants compare under isotropic Gaussian vs irregular blur degradations?

Key findings

• End-to-end DAN with alternating optimization substantially outperforms state-of-the-art two-step blind SR methods on synthetic data (notably surpassing IKC) and real images.
• DANv1 already outperforms IKC by 3.22 dB on Urban100 for scale 3, illustrating the value of end-to-end training.
• DANv2, featuring the Dual-Path Conditional Block (DPCB) and improved Estimator supervision, further improves results, with DANv2 surpassing DANv1 by 1.19 dB for scale 4.
• The dual-path design accelerates inference and stabilizes training, achieving higher speed and more robust kernel/HR estimation.
• Estimator is now supervised on complete kernels (not reduced-space), and a Softmax ensures kernel elements sum to one, improving kernel realism and convergence.
• The model effectively leverages information from both LR and SR images to estimate kernels, making the whole system more tolerant to estimation errors.