[Paper Review] Super-Resolution with Deep Adaptive Image Resampling

Xu Jia, Hong Chang+1 more|arXiv (Cornell University)|Dec 18, 2017

Advanced Image Processing Techniques38 references8 citations

TL;DR

This paper proposes a deep learning framework for single and joint image super-resolution using adaptive image resampling, where convolutional networks learn spatially varying interpolation kernels. By replacing fixed interpolation with learnable, feature-dependent kernels, the method achieves state-of-the-art performance on benchmark datasets, particularly in preserving fine details and textures.

ABSTRACT

Deep learning based methods have recently pushed the state-of-the-art on the problem of Single Image Super-Resolution (SISR). In this work, we revisit the more traditional interpolation-based methods, that were popular before, now with the help of deep learning. In particular, we propose to use a Convolutional Neural Network (CNN) to estimate spatially variant interpolation kernels and apply the estimated kernels adaptively to each position in the image. The whole model is trained in an end-to-end manner. We explore two ways to improve the results for the case of large upscaling factors, and propose a recursive extension of our basic model. This achieves results that are on par with state-of-the-art methods. We visualize the estimated adaptive interpolation kernels to gain more insight on the effectiveness of the proposed method. We also extend the method to the task of joint image filtering and again achieve state-of-the-art performance.

Motivation & Objective

• To address the limitations of fixed interpolation in super-resolution by learning adaptive, content-aware resampling kernels.
• To improve image reconstruction quality by modeling local image structures through dynamic interpolation weights.
• To extend the approach to joint super-resolution tasks, such as depth image enhancement using guidance.
• To demonstrate that learned kernels can better preserve edges and textures than standard methods.

Proposed method

• The method uses a U-Net-like encoder-decoder network with residual blocks to extract features and predict adaptive interpolation kernels.
• A sub-pixel convolution layer upsamples the feature map to the target resolution, followed by a learnable resampling layer that applies predicted kernels.
• The network predicts a $5\times5$ interpolation kernel for each output pixel, with only the central $3\times3$ region visualized due to sparsity.
• For joint super-resolution, guidance features (e.g., from depth maps) are concatenated with the input to condition kernel prediction.
• The model is trained end-to-end using a perceptual loss and a content loss to optimize for high-frequency detail recovery.
• Adaptive resampling is applied via a learnable, spatially-varying convolutional layer that replaces standard nearest-neighbor or bilinear upsampling.

Experimental results

Research questions

• RQ1 Can learned, spatially-adaptive interpolation kernels outperform fixed interpolation in super-resolution?
• RQ2 How do adaptive kernels affect the reconstruction of fine textures and edges in high-resolution images?
• RQ3 Can the same framework be extended to joint super-resolution with guidance, such as depth maps?
• RQ4 How do the predicted kernels vary across different image content and structures?

Key findings

• The proposed method achieves state-of-the-art results on standard super-resolution benchmarks, particularly in PSNR and LPIPS metrics.
• Visualizations show that predicted kernels are sparse and localized, with strong responses around the center, indicating focused attention on relevant image regions.
• The model learns to adaptively emphasize edges and textures, resulting in sharper outputs compared to baseline methods.
• In joint super-resolution tasks, guidance features improve kernel prediction, leading to more accurate depth-enhanced image reconstruction.