[Paper Review] ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolution

Axi Niu, Pham Xuan Trung+5 more|arXiv (Cornell University)|Jul 3, 2023

Advanced Image Processing TechniquesComputer Science3 citations

TL;DR

ACDMSR proposes a fast, conditional diffusion model for single image super-resolution by using a pre-trained super-resolution model's output as a conditional image, enabling high-quality, photo-realistic results. The method achieves state-of-the-art performance in both quantitative metrics and perceptual quality on benchmark datasets like Set5, Urban100, and BSD100, significantly outperforming prior diffusion-based and CNN-based approaches in visual fidelity while maintaining competitive PSNR and SSIM scores.

ABSTRACT

Diffusion models have gained significant popularity in the field of image-to-image translation. Previous efforts applying diffusion models to image super-resolution (SR) have demonstrated that iteratively refining pure Gaussian noise using a U-Net architecture trained on denoising at various noise levels can yield satisfactory high-resolution images from low-resolution inputs. However, this iterative refinement process comes with the drawback of low inference speed, which strongly limits its applications. To speed up inference and further enhance the performance, our research revisits diffusion models in image super-resolution and proposes a straightforward yet significant diffusion model-based super-resolution method called ACDMSR (accelerated conditional diffusion model for image super-resolution). Specifically, our method adapts the standard diffusion model to perform super-resolution through a deterministic iterative denoising process. Our study also highlights the effectiveness of using a pre-trained SR model to provide the conditional image of the given low-resolution (LR) image to achieve superior high-resolution results. We demonstrate that our method surpasses previous attempts in qualitative and quantitative results through extensive experiments conducted on benchmark datasets such as Set5, Set14, Urban100, BSD100, and Manga109. Moreover, our approach generates more visually realistic counterparts for low-resolution images, emphasizing its effectiveness in practical scenarios.

Motivation & Objective

• To address the slow inference speed of diffusion models in single image super-resolution (SISR), which limits practical deployment.
• To improve perceptual quality in SISR by leveraging conditional diffusion modeling beyond traditional PSNR-optimized methods.
• To investigate whether pre-super-resolved features from a pre-trained model can serve as effective conditionals for diffusion-based SISR.
• To develop a simple yet effective post-processing framework that enhances image realism without retraining the diffusion model from scratch.

Proposed method

• The method uses a pre-trained super-resolution model (e.g., RRDB, EDSR, RCAN, SwinIR) to generate a conditional image from the input low-resolution (LR) image.
• This conditional image is then used as a guide in a diffusion model that performs deterministic iterative denoising to refine the HR output.
• The diffusion model is trained to predict noise in the reverse denoising process, but the final output is generated using image-predicted loss, which improves perceptual quality.
• The framework is applied as a post-processing step to existing SR models, enabling fast inference with minimal architectural changes.
• The method supports various scale factors and is evaluated on multiple benchmarks including Set5, Set14, Urban100, BSD100, and Manga109.

Figure 1 : Illustration of our method. The model contains a stochastic forward diffusion process, gradually adding noise to an $\bm{I}^{HR}$ image. And a deterministic denoise process is applied to recover high-resolution and realistic images $\bm{I}^{SR}$ corresponding to $\bm{I}^{LR}$ images.

Experimental results

Research questions

• RQ1 Can using a pre-super-resolved image as a conditional input improve the quality of diffusion-based super-resolution?
• RQ2 How does the choice of conditional image (e.g., LR, EDSR, RCAN, SwinIR, or RRDB output) affect the final SR result?
• RQ3 Does training the diffusion model to predict the image rather than the noise lead to better perceptual quality?
• RQ4 Can a simple post-processing diffusion framework achieve state-of-the-art performance without retraining the main SR model?

Key findings

• On the Urban100 dataset at 4× scale, ACDMSR with SwinIR as the conditional model achieved a PSNR of 25.947 and SSIM of 0.6743, outperforming all baselines in both metrics.
• The LPIPS score was reduced to 0.0834 when using RCAN as the conditional model, indicating superior perceptual quality compared to other methods.
• The image-predicted loss variant outperformed the noise-predicted loss in both PSNR and LPIPS, confirming its superiority in learning image distribution for SR.
• Visual results on diverse content (e.g., text, animals, humans) showed that ACDMSR generates sharper textures and more realistic details than baseline methods.
• The method achieved state-of-the-art performance on Set5, Set14, BSD100, and Manga109, with consistent gains in perceptual quality despite slightly lower PSNR than some CNN-based models.

(a) PNSR