[Paper Review] Global Structure-Aware Diffusion Process for Low-Light Image Enhancement

Jinhui Hou, Zhiyu Zhu+4 more|arXiv (Cornell University)|Oct 26, 2023

Sparse and Compressive Sensing Techniques43 citations

TL;DR

A diffusion-based framework regularizes the ODE trajectory with global structure-aware and uncertainty-guided terms to improve low-light image enhancement, achieving state-of-the-art metrics on several LLIE benchmarks.

ABSTRACT

This paper studies a diffusion-based framework to address the low-light image enhancement problem. To harness the capabilities of diffusion models, we delve into this intricate process and advocate for the regularization of its inherent ODE-trajectory. To be specific, inspired by the recent research that low curvature ODE-trajectory results in a stable and effective diffusion process, we formulate a curvature regularization term anchored in the intrinsic non-local structures of image data, i.e., global structure-aware regularization, which gradually facilitates the preservation of complicated details and the augmentation of contrast during the diffusion process. This incorporation mitigates the adverse effects of noise and artifacts resulting from the diffusion process, leading to a more precise and flexible enhancement. To additionally promote learning in challenging regions, we introduce an uncertainty-guided regularization technique, which wisely relaxes constraints on the most extreme regions of the image. Experimental evaluations reveal that the proposed diffusion-based framework, complemented by rank-informed regularization, attains distinguished performance in low-light enhancement. The outcomes indicate substantial advancements in image quality, noise suppression, and contrast amplification in comparison with state-of-the-art methods. We believe this innovative approach will stimulate further exploration and advancement in low-light image processing, with potential implications for other applications of diffusion models. The code is publicly available at https://github.com/jinnh/GSAD.

Motivation & Objective

• Motivate and address limitations of pixel-wise regularization in diffusion-based LLIE methods.
• Regularize the diffusion ODE trajectory to preserve global image structure and details.
• Introduce non-local patch-based matrix rank regularization to capture global structures.
• Incorporate an uncertainty-guided mechanism to adapt regularization strength in challenging regions.
• Demonstrate improved restoration quality and robustness on standard LLIE datasets.

Proposed method

• Model the LLIE problem as a diffusion process conditioned on the input low-light image and learnable closed-form samples at each timestep.
• Regularize the reverse trajectory with a global structure-aware term via non-local, rank-based matrix representations of image patches across clusters, injected progressively during diffusion (κ_t scheduling).
• Construct learnable closed-form samples X_{t-1} from X_t to apply regularization on the learnable path rather than the fixed closed form, improving stability.
• Employ a non-local patch-based clustering of image blocks to form matrices whose ranks reflect global structures, and penalize divergence between current and ground-truth structures.
• Introduce an uncertainty map P_t via a pre-trained uncertainty model to weight the diffusion loss, emphasizing difficult regions.
• Optimize with a combined loss including the uncertainty-guided term and the structure-aware regularization term, with an adaptive training schedule.

Experimental results

Research questions

• RQ1 Does global structure-aware, rank-based regularization improve the curvature and stability of the diffusion reverse trajectory in LLIE?
• RQ2 Can non-local patch-based matrix rank modeling better preserve global textures and contrast compared with pixel-wise losses?
• RQ3 Does incorporating an uncertainty-guided regularization enhance learning in challenging low-light regions without sacrificing overall quality?
• RQ4 What is the impact of progressively injecting structure-aware regularization on LLIE performance across benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on LOLv1 and LOLv2 across PSNR, SSIM, and LPIPS, with the lowest LPIPS indicating superior perceptual quality.
• On LOLv1, the method records PSNR 27.839, SSIM 0.877, LPIPS 0.091; on LOLv2-real, PSNR 28.818, SSIM 0.895, LPIPS 0.095; on LOLv2-synthetic, PSNR 28.670, SSIM 0.944, LPIPS 0.047.
• The approach yields better NIQE scores on unpaired real-world LLIE datasets (DICM, LIME, MEF, NPE, VV) than competing methods, indicating stronger generalization.
• Ablation studies show that non-local rank-based regularization with adaptive scheduling, along with uncertainty-guided regularization, yields the largest gains in PSNR, SSIM, and LPIPS.
• Clustering with advanced hierarchical methods further improves PSNR and perceptual metrics compared to K-means, illustrating the importance of clustering choice for structure modeling.