[Paper Review] Low-Light Image Enhancement via Structure Modeling and Guidance

Xiaogang Xu, Ruixing Wang+1 more|arXiv (Cornell University)|May 10, 2023

Image Enhancement Techniques5 citations

TL;DR

This paper proposes a novel low-light image enhancement framework that jointly models structural edges and guides appearance enhancement using a GAN-trained structure-aware generative model and a structure-guided enhancement module. By leveraging edge maps for sharp, realistic results, the method achieves state-of-the-art performance across sRGB and RAW datasets with consistent improvements in PSNR and SSIM.

ABSTRACT

This paper proposes a new framework for low-light image enhancement by simultaneously conducting the appearance as well as structure modeling. It employs the structural feature to guide the appearance enhancement, leading to sharp and realistic results. The structure modeling in our framework is implemented as the edge detection in low-light images. It is achieved with a modified generative model via designing a structure-aware feature extractor and generator. The detected edge maps can accurately emphasize the essential structural information, and the edge prediction is robust towards the noises in dark areas. Moreover, to improve the appearance modeling, which is implemented with a simple U-Net, a novel structure-guided enhancement module is proposed with structure-guided feature synthesis layers. The appearance modeling, edge detector, and enhancement module can be trained end-to-end. The experiments are conducted on representative datasets (sRGB and RAW domains), showing that our model consistently achieves SOTA performance on all datasets with the same architecture.

Motivation & Objective

• Address the limitation of existing low-light enhancement methods that fail to explicitly model structural details, leading to blurry results and low SSIM.
• Improve reconstruction quality and structural fidelity in dark, noisy regions by modeling edges as structural priors.
• Develop a unified, end-to-end trainable framework that simultaneously enhances appearance and restores structural details.
• Demonstrate generalization and robustness across diverse datasets, including sRGB and RAW domains, with consistent SOTA performance.

Proposed method

• Design a structure-aware feature extractor (SAFE) that applies spatially-varying operations (adaptive long- and short-range computations) to extract structure-aware features from low-light images and their gradients.
• Train a modified StyleGAN-based generative model (S) with a GAN loss to predict edge maps, improving robustness to noise and low visibility in dark areas.
• Introduce a Structure-Guided Enhancement Module (SGEM) that uses spatially-adaptive convolutions and normalization parameters generated from predicted edge maps to refine appearance features.
• Implement the appearance modeling via a lightweight U-Net backbone, enhanced by the SGEM for residual refinement.
• Train the structure model (S), appearance model (A), and enhancement module (E) end-to-end using a combined loss combining GAN, L2, and perceptual losses.
• Enable generalization to other structural representations (e.g., segmentation, depth) by retraining S with corresponding ground truth maps from normal-light images.

Experimental results

Research questions

• RQ1 Can explicit structural modeling via edge prediction improve low-light image enhancement quality and reduce blurriness?
• RQ2 How does using a GAN loss for structure modeling in low-light images compare to traditional regression-based approaches in terms of artifact reduction and edge accuracy?
• RQ3 To what extent does structure-guided feature synthesis enhance appearance reconstruction beyond standard U-Net architectures?
• RQ4 Can the proposed framework generalize across different image domains (sRGB vs. RAW) and maintain SOTA performance with a single architecture?
• RQ5 How does incorporating additional supervised data (e.g., from AVA dataset) improve structure modeling in extremely dark regions?

Key findings

• The proposed method achieves SOTA performance on all benchmark datasets (SID, LOL-synthetic, LOL-real, and sRGB/RAW domains), with PSNR of 24.62 dB and SSIM of 0.867 on LOL-real, outperforming all baselines.
• The ablation study confirms that the GAN loss in structure modeling significantly reduces artifacts and improves edge map quality, especially in noisy, dark regions.
• The structure-guided enhancement module (SGEM) effectively improves appearance reconstruction, as shown by higher PSNR and SSIM scores compared to ablations without it.
• Using extra data from the AVA dataset to fine-tune the structure model (S) further improves performance, achieving PSNR of 25.57 dB and SSIM of 0.885 on LOL-real.
• User studies with 100 participants confirm that the method is perceived as superior in visual quality, receiving significantly more high ratings (4–5) and fewer low ratings (1–2) than baselines.
• The framework generalizes well to other structural representations (e.g., segmentation, depth maps), with results consistently outperforming baselines when using DPT-based supervision.