[Paper Review] Deep Back-Projection Networks for Single Image Super-resolution

Muhammad Haris, Greg Shakhnarovich+1 more|arXiv (Cornell University)|Apr 4, 2019

Advanced Image Processing Techniques13 references35 citations

TL;DR

This paper introduces Deep Back-Projection Networks (DBPN), which use iterative up- and down-sampling projection units with error feedback to improve single image super-resolution, achieving state-of-the-art results on multiple datasets, especially at large scales such as 8x.

ABSTRACT

Previous feed-forward architectures of recently proposed deep super-resolution networks learn the features of low-resolution inputs and the non-linear mapping from those to a high-resolution output. However, this approach does not fully address the mutual dependencies of low- and high-resolution images. We propose Deep Back-Projection Networks (DBPN), the winner of two image super-resolution challenges (NTIRE2018 and PIRM2018), that exploit iterative up- and down-sampling layers. These layers are formed as a unit providing an error feedback mechanism for projection errors. We construct mutually-connected up- and down-sampling units each of which represents different types of low- and high-resolution components. We also show that extending this idea to demonstrate a new insight towards more efficient network design substantially, such as parameter sharing on the projection module and transition layer on projection step. The experimental results yield superior results and in particular establishing new state-of-the-art results across multiple data sets, especially for large scaling factors such as 8x.

Motivation & Objective

• Motivate improved LR-to-HR learning by leveraging mutual dependencies between low- and high-resolution representations.
• Develop an end-to-end trainable architecture with iterative up- and down-projection units.
• Introduce error feedback to refine HR features via projection errors.
• Explore network design enhancements such as parameter sharing and projection-step transitions for efficiency.
• Show competitive or state-of-the-art performance on large scaling factors (e.g., 8x) across standard SR benchmarks.

Proposed method

• Propose iterative up- and down-projection units that map LR features to HR and back, forming an error-correcting feedback loop.
• Use deep concatenation to fuse HR feature maps from all up- projection units for reconstruction.
• Define up- and down-projection units with residual learning-like mechanisms and large receptive fields (8x8, 12x12 kernels) to capture HR/LR relationships.
• Incorporate dense connections (Dense DBPN) to reuse features and improve learning without normalization/batch norm.
• Introduce recurrent variants (shared projections, transition layers) to reduce parameters and widen receptive fields.
• Experiment with residual DBPN to start from an interpolated LR image and add the reconstruction residual at the end.

Experimental results

Research questions

• RQ1 Can iterative mutual LR-HR projections with error feedback improve SR quality, particularly for large scale factors?
• RQ2 Do architectural variants (dense projections, recurrent/transition layers, residual learning) yield better accuracy and parameter efficiency compared to baseline DBPN?
• RQ3 What is the impact of feature concatenation across projection units on SR reconstruction quality?
• RQ4 How does error feedback (EF) affect PSNR and visual sharpness across standard SR benchmarks?

Key findings

• DBPN achieves state-of-the-art performance on NTIRE2018 and PIRM2018 SR challenges.
• DBPN variants show improved PSNR/SSIM over several baselines on Set5, Set14, BSDS100, Urban100, and Manga109 for 4x SR, with notable gains at 8x.
• Dense DBPN (dense projection units) improves performance over non-dense variants, especially on challenging datasets.
• Recurrent and residual designs (DBPN-RES-MR64-3) provide strong results with fewer parameters than some baselines, achieving 32.65 dB (Set5) and 29.03 dB (Set14) for 4x, and 31.74 dB on Manga109 (4x).
• Error feedback (EF) substantially improves SR quality, e.g., 0.53 dB (Set5) and 0.26 dB (Set14) gains for DBPN-S with EF over without EF.
• The best-performing configuration (DBPN-RES-MR64-3) achieves competitive results with fewer parameters than several prior methods, highlighting efficiency gains from the proposed design.