[Paper Review] SinSR: Diffusion-Based Image Super-Resolution in a Single Step

Yufei Wang, Wenhan Yang+8 more|arXiv (Cornell University)|Nov 23, 2023

Advanced Image Processing Techniques4 citations

TL;DR

SinSR proposes a single-step diffusion-based super-resolution method that distills a deterministic mapping from a pre-trained teacher model, enabling high-fidelity image generation in just one inference step. By leveraging a novel consistency-preserving loss and a derived deterministic sampling strategy, SinSR achieves performance on par or superior to state-of-the-art methods with up to a \times10 speedup.

ABSTRACT

While super-resolution (SR) methods based on diffusion models exhibit promising results, their practical application is hindered by the substantial number of required inference steps. Recent methods utilize degraded images in the initial state, thereby shortening the Markov chain. Nevertheless, these solutions either rely on a precise formulation of the degradation process or still necessitate a relatively lengthy generation path (e.g., 15 iterations). To enhance inference speed, we propose a simple yet effective method for achieving single-step SR generation, named SinSR. Specifically, we first derive a deterministic sampling process from the most recent state-of-the-art (SOTA) method for accelerating diffusion-based SR. This allows the mapping between the input random noise and the generated high-resolution image to be obtained in a reduced and acceptable number of inference steps during training. We show that this deterministic mapping can be distilled into a student model that performs SR within only one inference step. Additionally, we propose a novel consistency-preserving loss to simultaneously leverage the ground-truth image during the distillation process, ensuring that the performance of the student model is not solely bound by the feature manifold of the teacher model, resulting in further performance improvement. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method can achieve comparable or even superior performance compared to both previous SOTA methods and the teacher model, in just one sampling step, resulting in a remarkable up to x10 speedup for inference. Our code will be released at https://github.com/wyf0912/SinSR

Motivation & Objective

• To address the high inference cost of diffusion-based super-resolution models, which typically require 15+ steps for high-quality results.
• To enable single-step generation without sacrificing perceptual quality or diversity.
• To reduce training and inference overhead while maintaining high-fidelity image reconstruction.
• To improve generalization by directly incorporating ground-truth supervision during distillation, beyond just knowledge distillation from the teacher model.

Proposed method

• Derives a deterministic sampling strategy from the recent SOTA method ResShift to accelerate the generation of paired training data between input noise and high-resolution images.
• Trains a student model to learn a bi-directional deterministic mapping between input random noise and the corresponding high-resolution output.
• Introduces a consistency-preserving loss that minimizes the difference between the ground-truth image and the image reconstructed from the predicted initial noise state.
• Uses knowledge distillation to transfer the teacher model's generative capability into the student model, enabling one-step inference.
• Initializes the student model from the pre-trained teacher model to reduce fine-tuning cost and improve convergence.
• Employs ODE solving during training to generate high-quality training pairs, while enabling extremely fast inference in a single step.

Experimental results

Research questions

• RQ1 Can a diffusion-based super-resolution model achieve high perceptual quality in a single inference step without relying on lengthy Markov chains?
• RQ2 Is it feasible to distill a deterministic mapping from a teacher diffusion model to a student model for one-step image generation?
• RQ3 Can incorporating ground-truth images during distillation improve the student model’s performance beyond what is achievable through knowledge distillation alone?
• RQ4 Does a consistency-preserving loss lead to better fidelity and reduced artifacts compared to standard inversion-based methods like DDIM?
• RQ5 How does the proposed method compare in speed and performance to existing SOTA diffusion-based SR methods?

Key findings

• SinSR achieves comparable or superior performance to the teacher model ResShift [45] in just one inference step, with a maximum of \times10 speedup.
• On the RealSet65 dataset, SinSR (24.32M parameters) achieves a CLIPIQA score of 0.6499 and MUSIQ score of 58.71, outperforming the smaller ResShift model (24.32M) with 52.71 MUSIQ.
• The consistency-preserving loss enables the student model to achieve better fidelity than DDIM inversion, as shown in visual comparisons with improved detail reconstruction.
• The training time for SinSR is reduced to ~2.57 days (30k iterations) compared to ~7.64 days (500k iterations) for ResShift, despite using a more complex loss.
• The student model trained to learn the deterministic mapping between noise and HR image is significantly more robust to model size reduction than the denoising-based approach, indicating easier optimization.
• Even with a simpler distillation strategy, SinSR outperforms more complex methods like Rectified Flow, which suffer from error accumulation during iterative distillation.