[Paper Review] Implicit Diffusion Models for Continuous Super-Resolution

Sicheng Gao, Xuhui Liu+7 more|arXiv (Cornell University)|Mar 29, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes an Implicit Diffusion Model (IDM) for continuous image super-resolution that combines denoising diffusion with implicit neural representations to generate high-fidelity, resolution-continuous results. By introducing a scale-adaptive conditioning mechanism with a learnable scaling factor, IDM dynamically balances low-resolution input features and generated details, enabling end-to-end training without fixed magnification constraints and achieving state-of-the-art performance on natural and facial image SR benchmarks.

ABSTRACT

Image super-resolution (SR) has attracted increasing attention due to its wide applications. However, current SR methods generally suffer from over-smoothing and artifacts, and most work only with fixed magnifications. This paper introduces an Implicit Diffusion Model (IDM) for high-fidelity continuous image super-resolution. IDM integrates an implicit neural representation and a denoising diffusion model in a unified end-to-end framework, where the implicit neural representation is adopted in the decoding process to learn continuous-resolution representation. Furthermore, we design a scale-controllable conditioning mechanism that consists of a low-resolution (LR) conditioning network and a scaling factor. The scaling factor regulates the resolution and accordingly modulates the proportion of the LR information and generated features in the final output, which enables the model to accommodate the continuous-resolution requirement. Extensive experiments validate the effectiveness of our IDM and demonstrate its superior performance over prior arts.

Motivation & Objective

• To address the limitations of existing super-resolution models that produce over-smoothed outputs and are restricted to fixed magnifications.
• To enable high-fidelity, continuous-resolution image super-resolution without requiring retraining or cascaded architectures.
• To integrate implicit neural representations with diffusion models for resolution-continuous feature learning.
• To design a scale-adaptive conditioning mechanism that dynamically modulates the contribution of low-resolution features and generated details during inference.
• To achieve state-of-the-art performance in both quantitative metrics and perceptual quality across diverse magnifications.

Proposed method

• The model uses a denoising diffusion process as the core generative mechanism, iteratively refining a latent representation toward a high-resolution image.
• An implicit neural representation is embedded in the U-Net decoder, parameterized by a coordinate-based MLP to model images as continuous functions of spatial coordinates.
• A scale-adaptive conditioning mechanism is introduced, consisting of a low-resolution conditioning network and a learnable scaling factor to control resolution output dynamically.
• The scaling factor modulates the feature fusion between encoded low-resolution features and generated denoised features via an adaptive MLP, enabling continuous resolution control.
• The entire framework is trained in an end-to-end manner without additional priors or two-stage training, unlike prior generative SR methods.
• The model is trained on a range of magnifications (e.g., 8×) and generalizes to both in-distribution and out-of-distribution scales during inference.

Experimental results

Research questions

• RQ1 Can a diffusion-based model achieve high-fidelity, continuous-resolution image super-resolution without being constrained to fixed magnifications?
• RQ2 How can implicit neural representations be effectively integrated into a diffusion model to enable resolution-continuous generation?
• RQ3 What is the role of a learnable scaling factor in dynamically balancing low-resolution priors and generated high-frequency details?
• RQ4 Can an end-to-end, prior-free framework outperform existing two-stage or prior-augmented generative SR models in both fidelity and generalization?
• RQ5 How does the model perform on out-of-distribution magnifications beyond the training range?

Key findings

• IDM achieves state-of-the-art performance on both natural image and facial image super-resolution benchmarks, outperforming prior arts in PSNR and SSIM, with a 0.50 dB improvement in PSNR and 0.03 in SSIM over previous generative methods.
• On the CelebA-HQ dataset, IDM achieves superior LPIPS scores compared to LIIF and SR3, indicating better perceptual consistency even when PSNR is lower.
• IDM generalizes effectively to out-of-distribution magnifications (e.g., 9× and 10×) when trained on 8× SR, maintaining high perceptual quality and structural fidelity.
• Abllation studies confirm that the scaling factor is critical for controlling the balance between low-resolution priors and generated details, with performance degrading significantly when omitted or replaced with naive conditioning.
• The proposed LR conditioning network outperforms both direct upsampling and encoder-based conditioning, producing sharper textures and more accurate facial details in qualitative comparisons.
• Despite using only 800 training images (vs. 2800 in other methods), IDM achieves better performance, demonstrating data efficiency and robustness.