[Paper Review] PSENet: Progressive Self-Enhancement Network for Unsupervised Extreme-Light Image Enhancement

Hue Minh Thi Nguyen, Diep Tran+2 more|arXiv (Cornell University)|Oct 3, 2022

Image Enhancement Techniques4 citations

TL;DR

PSENet proposes a novel unsupervised deep learning framework for extreme-light image enhancement that generates pseudo-ground-truth images through progressive self-enhancement using multiple exposure-simulated sources. By iteratively selecting the best-performing image regions based on perceptual quality scores—well-exposedness, local contrast, and color saturation—it achieves state-of-the-art performance on both under- and over-exposed images without requiring any real ground-truth images.

ABSTRACT

The extremes of lighting (e.g. too much or too little light) usually cause many troubles for machine and human vision. Many recent works have mainly focused on under-exposure cases where images are often captured in low-light conditions (e.g. nighttime) and achieved promising results for enhancing the quality of images. However, they are inferior to handling images under over-exposure. To mitigate this limitation, we propose a novel unsupervised enhancement framework which is robust against various lighting conditions while does not require any well-exposed images to serve as the ground-truths. Our main concept is to construct pseudo-ground-truth images synthesized from multiple source images that simulate all potential exposure scenarios to train the enhancement network. Our extensive experiments show that the proposed approach consistently outperforms the current state-of-the-art unsupervised counterparts in several public datasets in terms of both quantitative metrics and qualitative results. Our code is available at https://github.com/VinAIResearch/PSENet-Image-Enhancement.

Motivation & Objective

• Address the lack of effective unsupervised methods for enhancing both under- and over-exposed images, which are common in real-world photography.
• Overcome the limitations of existing unsupervised methods that fail on over-exposure by introducing a self-supervised pseudo-GT generation mechanism.
• Eliminate the need for expensive, paired real-world well-exposed images by synthesizing pseudo-GT from multiple exposure variants of the same input.
• Improve image quality through progressive refinement, ensuring each epoch's output is at least as good as the previous one.
• Develop a perceptually driven, non-reference quality score to guide the selection of optimal image regions for pseudo-GT construction.

Proposed method

• Construct pseudo-ground-truth images by combining regions from four sources: the input image, its gamma-adjusted brighter/darker versions, and the network's own output from the previous training epoch.
• Use a non-reference perceptual quality score based on well-exposedness, local contrast, and color saturation to evaluate and select the best image regions across sources.
• Apply a region-wise comparison and fusion strategy (RCS) to progressively refine the pseudo-GT, ensuring improvement across training epochs.
• Train an end-to-end enhancement network using the generated pseudo-GT, enabling self-supervised learning without any paired real GT images.
• Optimize the network with a well-exposedness level hyperparameter μ, which controls the target brightness range and balances performance on under- and over-exposed cases.
• Integrate the pseudo-GT generation into a progressive training loop, where the network’s own output from the prior epoch becomes a key source for improving the next iteration.

Figure 1: Visual comparison on an over-exposed scene. Most of the previous state-of-the-art failed to recover the over-exposed case except recent work proposed by Afifi et al . [ 1 ] , which is trained with full supervision.

Experimental results

Research questions

• RQ1 Can a self-supervised framework effectively enhance both under- and over-exposed images without relying on real ground-truth images?
• RQ2 How effective is a non-reference perceptual quality score based on well-exposedness, contrast, and saturation in guiding pseudo-GT generation?
• RQ3 Does progressive refinement using the network’s own previous outputs lead to consistent improvement in image enhancement quality?
• RQ4 How does the choice of well-exposedness level μ affect the model’s ability to generalize across extreme lighting conditions?
• RQ5 Can the proposed method outperform existing unsupervised methods on benchmark datasets like SICE and Afifi, despite lacking paired supervision?

Key findings

• PSENet achieves state-of-the-art performance on the SICE dataset, with a PSNR of 17.74 and SSIM of 0.704 when using the default network width (# channels ×1).
• On the Afifi dataset, PSENet achieves a PSNR of 19.36 and SSIM of 0.869, outperforming all unsupervised baselines and approaching supervised SOTA performance.
• The model trained with a well-exposedness level μ = 0.5 achieves the best balance between under- and over-exposed image recovery, yielding the highest quantitative scores.
• Visual comparisons show that PSENet produces images with better contrast, color preservation, and detail recovery than ZeroDCE, EnlightenGAN, and Zheng and Gupta [45], especially in challenging regions like shadows and highlights.
• Ablation studies confirm that the proposed quality score and region combination strategy (RCS) significantly outperform prior methods, with a 1.64 PSNR gain over the baseline from [25].
• The model remains robust across different network widths, with minimal performance drop when scaling parameters, indicating stable and generalizable optimization.

Figure 2: Overview of our proposed framework which comprises three main modules: reference image generator, pseudo GT image generator, and enhancement network. Given the input image $I$ , the reference image generator randomly generates multiple reference images with different exposure values, half