[Paper Review] Difficulty-aware Image Super Resolution via Deep Adaptive Dual-Network

Jinghui Qin, Ziwei Xie+2 more|arXiv (Cornell University)|Apr 11, 2019

Advanced Image Processing Techniques18 references4 citations

TL;DR

This paper proposes a difficulty-aware image super-resolution method using a deep adaptive dual-network framework that separately processes easy and hard image regions via two specialized branches: a plain branch (PB) for smooth regions and a complex branch (CB) for textured regions. By leveraging a PSNR-based difficulty recognition network to generate adaptive region masks, the model achieves state-of-the-art performance on standard benchmarks, outperforming prior methods by up to 0.50 dB in PSNR on Urban100 under ×2 scaling.

ABSTRACT

Recently, deep learning based single image super-resolution(SR) approaches have achieved great development. The state-of-the-art SR methods usually adopt a feed-forward pipeline to establish a non-linear mapping between low-res(LR) and high-res(HR) images. However, due to treating all image regions equally without considering the difficulty diversity, these approaches meet an upper bound for optimization. To address this issue, we propose a novel SR approach that discriminately processes each image region within an image by its difficulty. Specifically, we propose a dual-way SR network that one way is trained to focus on easy image regions and another is trained to handle hard image regions. To identify whether a region is easy or hard, we propose a novel image difficulty recognition network based on PSNR prior. Our SR approach that uses the region mask to adaptively enforce the dual-way SR network yields superior results. Extensive experiments on several standard benchmarks (e.g., Set5, Set14, BSD100, and Urban100) show that our approach achieves state-of-the-art performance.

Motivation & Objective

• Address the limitation of existing single-model super-resolution networks that treat all image regions uniformly, despite varying reconstruction difficulty.
• Recognize and categorize image regions based on their intrinsic super-resolution difficulty to enable tailored processing.
• Design a dual-branch network architecture where one branch specializes in easy (plain) regions and another in hard (complex) regions.
• Improve overall super-resolution performance by adaptively routing regions to the most suitable branch using a learned mask.
• Demonstrate that region-level difficulty-aware processing leads to superior restoration quality compared to unified models.

Proposed method

• Introduce a difficulty identifier module (DIM) that uses PSNR prior from bicubic interpolation to classify image patches as easy or hard.
• Train a separate plain branch (PB) and complex branch (CB) in a dual-way SR network: PB optimized for smooth/easy regions, CB for textured/hard regions.
• Use a mask generator to produce spatial masks that assign each image patch to the most appropriate branch based on its difficulty.
• Apply the adaptive mask during inference to route each region to the corresponding branch, enabling selective, optimal processing.
• Train the entire framework end-to-end with a combined loss function that combines reconstruction and perceptual losses.
• Leverage PSNR values between bicubic-interpolated LR patches and their HR counterparts as a proxy for difficulty, enabling weakly supervised difficulty estimation.

Fig. 1 : Qualitative comparison with EDSR and SRCNN. EDSR reconstructs more clearly on the complex/hard region than SRCNN, but it reconstructs worse on the smooth/plain region than SRCNN. This reveals that it is suboptimal to incorporate a single model to process all regions.

Experimental results

Research questions

• RQ1 Can modeling image super-resolution as a difficulty-aware, region-specific task improve reconstruction quality beyond uniform processing?
• RQ2 How effective is PSNR-based prior in identifying hard vs. easy image regions for super-resolution?
• RQ3 Does a dual-branch network architecture, where each branch specializes in a difficulty class, outperform a single unified model?
• RQ4 Can adaptive routing of image regions to specialized branches lead to better PSNR and SSIM scores on standard benchmarks?
• RQ5 What is the impact of separate training for PB and CB on performance across diverse image datasets with mixed difficulty?

Key findings

• The proposed method achieves state-of-the-art PSNR performance, improving by up to 0.50 dB on Urban100 under ×2 scaling compared to prior methods.
• On Set14, the model achieves a PSNR gain of 0.46 dB over the baseline EDSR under ×2 scaling, demonstrating consistent improvement.
• The dual-branch adaptive approach (PB + CB) outperforms both individual branches, with PSNR improvements of 0.30 dB and 0.36 dB over MemNet on Urban100 under ×4 scaling.
• On hard image patches, the complex branch (CB) achieves significantly higher PSNR (e.g., 33.06 dB at ×2) than the plain branch (PB), confirming its specialization.
• On easy image patches, the plain branch (PB) achieves superior results (e.g., 53.95 dB at ×2), indicating its strength in smooth regions.
• Despite additional computational cost from the difficulty module, the full model remains competitive in inference speed and parameter efficiency compared to state-of-the-art methods like IDN and LapSRN.

Fig. 2 : Overview of our dual-way SR framework. Our framework consists of three key components: difficulty identifier module(DIM), mask generator and a dual-way SR network. Our dual-way SR network consists of two branches: a complex branch(CB) and a plain branch(PB). The CB is used to restore comple