[Paper Review] Hierarchical Integration Diffusion Model for Realistic Image Deblurring
HI-Diff integrates a diffusion model in a highly compressed latent space and fuses its prior via a hierarchical integration module into a regression-based Transformer, achieving state-of-the-art deblurring especially in complex blur and real-world datasets.
Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly in terms of details reconstruction. However, the diffusion model requires a large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion-based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in a highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by a regression-based method to obtain better distortion accuracy. Meanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-based model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods. Code and trained models are available at https://github.com/zhengchen1999/HI-Diff.
Motivation & Objective
- Motivate improving image deblurring under complex, non-uniform blur with better detail preservation.
- Leverage diffusion models to generate informative priors while maintaining distortion accuracy.
- Enhance generalization across blurry scenarios via hierarchical multi-scale fusion with a regression-based backbone.
Proposed method
- Use a two-stage training framework to learn a highly compact latent representation for prior features.
- Stage 1: latent encoder compresses GT images into a compact prior z and integrates it into a Restormer-based Transformer via a Hierarchical Integration Module (HIM).
- Stage 2: train a latent diffusion model to generate the prior feature z from Gaussian noise, conditioning on blur-relevant latent codes, and jointly train the diffusion model with the Transformer through HIM.
- Hierarchical Integration Module (HIM) performs cross-attention between Transformer intermediate features and multi-scale prior features z1, z2, z3 to fuse information from multiple scales.
- The latent diffusion model operates in a highly compact latent space (N=16 tokens, C'=256) to reduce computational load.
- The training objective combines a deblurring loss (L1 between deblurred and GT) and a diffusion loss (L1 between hat{z} and z) to align diffusion priors with the regression backbone.
Experimental results
Research questions
- RQ1How can diffusion priors be effectively integrated into a regression-based deblurring model to improve detail reconstruction without compromising distortion-based metrics?
- RQ2Does operating diffusion models in a highly compressed latent space with hierarchical, multi-scale fusion improve generalization to real-world blur?
- RQ3What is the impact of joint training of diffusion priors and the Transformer on deblurring performance and stability?
- RQ4How does the number of diffusion iterations in latent space affect priors and final deblurring quality?
Key findings
- HI-Diff achieves state-of-the-art PSNR/SSIM on GoPro, HIDE, and RealBlur datasets compared to 16 baselines.
- Incorporating diffusion priors in latent space yields up to about 0.28 dB PSNR improvement over a baseline Restormer.
- Hierarchical multi-scale prior integration significantly outperforms single-scale priors (0.24 dB PSNR drop when using single-scale).
- Joint training of diffusion model and Transformer substantially enhances performance vs. training diffusion alone (about 1.51 dB PSNR improvement).
- Only a small number of diffusion iterations (T around 8) suffices in latent space due to low-dimensional priors.
- HI-Diff shows strong generalization to RealBlur-R/J and RWBI after GoPro training, outperforming several state-of-the-art methods.
Read this paper
Open the full text in the Nubint reader and ask about any passage as you read.
Read this paper on NubintThis review was generated by AI from the original paper. Please check the paper before citing it.