[Paper Review] Learning to Kindle the Starlight

Yu Yuan, Jiaqi Wu+5 more|arXiv (Cornell University)|Nov 16, 2022

Image Enhancement Techniques4 citations

TL;DR

This paper introduces StarDiffusion, the first conditional denoising diffusion probabilistic model (DDPM) for star field image enhancement, leveraging a novel dynamic stochastic corruption strategy to improve performance on the newly proposed Star Field Image Enhancement Benchmark (SFIEB). The method achieves state-of-the-art results in both perceptual quality and quantitative metrics, significantly outperforming existing low-light image enhancement techniques on star field images.

ABSTRACT

Capturing highly appreciated star field images is extremely challenging due to light pollution, the requirements of specialized hardware, and the high level of photographic skills needed. Deep learning-based techniques have achieved remarkable results in low-light image enhancement (LLIE) but have not been widely applied to star field image enhancement due to the lack of training data. To address this problem, we construct the first Star Field Image Enhancement Benchmark (SFIEB) that contains 355 real-shot and 854 semi-synthetic star field images, all having the corresponding reference images. Using the presented dataset, we propose the first star field image enhancement approach, namely StarDiffusion, based on conditional denoising diffusion probabilistic models (DDPM). We introduce dynamic stochastic corruptions to the inputs of conditional DDPM to improve the performance and generalization of the network on our small-scale dataset. Experiments show promising results of our method, which outperforms state-of-the-art low-light image enhancement algorithms. The dataset and codes will be open-sourced.

Motivation & Objective

• To address the lack of training data for deep learning-based star field image enhancement.
• To develop a dedicated enhancement method tailored for star field images, which differ significantly from general low-light images due to sparse, high-contrast stars and large landscapes.
• To construct the first comprehensive benchmark, SFIEB, with 355 real-shot and 854 semi-synthetic star field image pairs for evaluation.
• To improve generalization and performance on small-scale datasets by introducing dynamic stochastic corruptions in a conditional DDPM framework.

Proposed method

• Proposes StarDiffusion, a conditional denoising diffusion probabilistic model (DDPM) specifically designed for star field image enhancement.
• Applies dynamic stochastic corruptions to the input images during the reverse denoising process to enhance model generalization and robustness on the small SFIEB dataset.
• Uses a conditional DDPM architecture where the reference image is concatenated with the noisy input at each denoising step to guide the generation process.
• Employs a two-stage training strategy: first pre-training on general low-light datasets (e.g., LOL), then fine-tuning on the SFIEB for domain-specific adaptation.
• Introduces a data augmentation strategy using semi-synthetic images generated from real star field images with controlled degradation to expand training diversity.
• Leverages the SFIEB benchmark to enable fair comparison and evaluation of star field enhancement methods.

Experimental results

Research questions

• RQ1 Can a diffusion-based model achieve superior performance in enhancing star field images compared to existing low-light image enhancement methods?
• RQ2 How effective is dynamic stochastic corruption in improving the generalization of a conditional DDPM on a small-scale, specialized dataset like SFIEB?
• RQ3 To what extent can a model pre-trained on general low-light data be fine-tuned on SFIEB to achieve high-quality star field enhancement?
• RQ4 Does the proposed StarDiffusion method generalize well to real-world star field images captured by consumer-grade devices?
• RQ5 Can the SFIEB benchmark serve as a reliable and effective evaluation platform for future star field image enhancement research?

Key findings

• StarDiffusion achieves a PSNR of 22.7895 and SSIM of 0.8073 on the SFIEB test set, significantly outperforming all baseline LLIE methods.
• In user studies, StarDiffusion received the highest average perception score (4.6/5), with users rating it as having the most natural colors and least noise.
• After fine-tuning on SFIEB, RetinexNet and Zero-DCE showed improved performance but still underperformed StarDiffusion, confirming its superiority on the star field task.
• StarDiffusion trained on SFIEB alone achieved competitive PSNR (18.3263) on the LOL benchmark, indicating strong generalization to general low-light images.
• The method effectively enhances star field images from consumer devices (e.g., iPhone12, GoPro, Sony α6400), producing visually superior results with clearer stars and better contrast.
• The ablation study confirms that dynamic stochastic corruptions significantly improve model generalization and performance on the small SFIEB dataset.