[Paper Review] Learning a Single Convolutional Super-Resolution Network for Multiple Degradations

Kai Zhang, Wangmeng Zuo+1 more|arXiv (Cornell University)|Dec 17, 2017

Advanced Image Processing Techniques47 references58 citations

TL;DR

This paper proposes SRMD, a single CNN that handles multiple and spatially variant degradations for image super-resolution by incorporating degradation information (blur kernel and noise level) as input through dimensionality stretching. It trains on synthetic degradations and achieves competitive results while being fast and scalable.

ABSTRACT

Recent years have witnessed the unprecedented success of deep convolutional neural networks (CNNs) in single image super-resolution (SISR). However, existing CNN-based SISR methods mostly assume that a low-resolution (LR) image is bicubicly downsampled from a high-resolution (HR) image, thus inevitably giving rise to poor performance when the true degradation does not follow this assumption. Moreover, they lack scalability in learning a single model to non-blindly deal with multiple degradations. To address these issues, we propose a general framework with dimensionality stretching strategy that enables a single convolutional super-resolution network to take two key factors of the SISR degradation process, i.e., blur kernel and noise level, as input. Consequently, the super-resolver can handle multiple and even spatially variant degradations, which significantly improves the practicability. Extensive experimental results on synthetic and real LR images show that the proposed convolutional super-resolution network not only can produce favorable results on multiple degradations but also is computationally efficient, providing a highly effective and scalable solution to practical SISR applications.

Motivation & Objective

• Address the practical limitation of SISR models trained for a single bicubic degradation by enabling a single model to handle multiple and spatially variant degradations.
• Introduce a degradation-aware CNN framework that jointly accounts for blur and noise during SR.
• Develop a simple, scalable training strategy using synthetic degradations to achieve real-world applicability.
• Provide insights into MAP-based interpretation of CNNs for SR and justify the design choices

Proposed method

• Adopt a degradation-aware CNN by concatenating the LR image with degradation maps that encode blur kernel and noise level through a dimensionality stretching strategy.
• Vectorize the blur kernel, project it with PCA to a t-dimensional space, and form degradation maps of size W x H x (t+1) together with the noise level.
• Use a plain CNN with 12 layers of 3x3 filters, ReLU, and Batch Normalization, plus a final sub-pixel convolution layer to generate the HR image.
• Train separate models for scale factors 2, 3, and 4, including a noise-free variant SRMDNF obtained by fine-tuning SRMD.
• Synthesize training data by applying isotropic/anisotropic Gaussian blur, bicubic downsampling, and AWGN to a large clean HR image set.
• Formulate the SR problem under a MAP-inspired view to motivate jointly learning data fidelity and priors, enabling a single model to handle multiple degradations via degradation inputs.

Experimental results

Research questions

• RQ1 Can a single model effectively handle multiple and spatially variant degradations in SISR?
• RQ2 Is it feasible to train a practical, degradation-aware SR model using synthetic data that transfers to real LR images?
• RQ3 Does incorporating blur kernel and noise level as inputs improve SR performance over bicubic-degradation-based models?
• RQ4 How does a degradation-aware CNN compare with model-based optimization and other CNN-based SR methods under general and spatially variant degradations?

Key findings

• SRMD achieves competitive PSNR/SSIM on standard bicubic benchmarks, closely matching or exceeding state-of-the-art CNN-based SR methods for scale factors 2, 3, and 4 (e.g., SRMD vs VDSR: comparable at scale 2; SRMD outperforms at larger scales; SRMDNF yields best overall PSNR/SSIM on several sets).
• SRMDNF (noise-free) often provides the best overall quantitative results, e.g., on Set5 with scale 4, SRMDNF reaches 31.96 in PSNR/SSIM 0.893 (from Table 1).
• The method demonstrates strong generalization to general degradations beyond bicubic, outperforming NCSR and IRCNN and surpassing cascaded DnCNN+SRMDNF in PSNR gains as blur width increases.
• SRMD works well for spatially variant degradations as evidenced by qualitative results showing recovered edges and textures under varying blur/noise maps.
• The approach provides practical performance with fast runtime (e.g., SRMD reconstructs 1024x1024 images in 0.027–0.084 seconds depending on scale).
• Experiments on real LR images show SRMD delivering more visually plausible HR results and better edge preservation than competing methods like VDSR and Waifu2x.