[Paper Review] Diving into Darkness: A Dual-Modulated Framework for High-Fidelity Super-Resolution in Ultra-Dark Environments

Jiaxin Gao, Ziyu Yue+4 more|arXiv (Cornell University)|Sep 11, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes UltraBM, a dual-modulated framework for high-fidelity super-resolution in ultra-dark environments, integrating a self-regularized luminance constraint and Illuminance-Semantic Dual Modulation (ISDM) to preserve illumination and color details. It introduces a Resolution-Sensitive Merging Up-sampler (RSMU) that reduces artifacts and halos, achieving a 5% PSNR gain and 43% LPIPS improvement over SOTA methods, with 19× higher RMSE generalization across darkness levels.

ABSTRACT

Super-resolution tasks oriented to images captured in ultra-dark environments is a practical yet challenging problem that has received little attention. Due to uneven illumination and low signal-to-noise ratio in dark environments, a multitude of problems such as lack of detail and color distortion may be magnified in the super-resolution process compared to normal-lighting environments. Consequently, conventional low-light enhancement or super-resolution methods, whether applied individually or in a cascaded manner for such problem, often encounter limitations in recovering luminance, color fidelity, and intricate details. To conquer these issues, this paper proposes a specialized dual-modulated learning framework that, for the first time, attempts to deeply dissect the nature of the low-light super-resolution task. Leveraging natural image color characteristics, we introduce a self-regularized luminance constraint as a prior for addressing uneven lighting. Expanding on this, we develop Illuminance-Semantic Dual Modulation (ISDM) components to enhance feature-level preservation of illumination and color details. Besides, instead of deploying naive up-sampling strategies, we design the Resolution-Sensitive Merging Up-sampler (RSMU) module that brings together different sampling modalities as substrates, effectively mitigating the presence of artifacts and halos. Comprehensive experiments showcases the applicability and generalizability of our approach to diverse and challenging ultra-low-light conditions, outperforming state-of-the-art methods with a notable improvement (i.e., $\uparrow$5\% in PSNR, and $\uparrow$43\% in LPIPS). Especially noteworthy is the 19-fold increase in the RMSE score, underscoring our method's exceptional generalization across different darkness levels. The code will be available online upon publication of the paper.

Motivation & Objective

• Address the underexplored challenge of super-resolution in ultra-dark environments where low signal-to-noise ratio and uneven illumination degrade image quality.
• Overcome limitations of cascaded low-light enhancement and super-resolution methods that fail to preserve color fidelity and fine details.
• Develop a unified framework that simultaneously enhances brightness and magnifies resolution while minimizing artifacts and halos.
• Introduce a novel dual-stream learning architecture to model illumination and semantic color priors for improved feature-level preservation.
• Ensure robust generalization across diverse ultra-low-light conditions, including extreme exposure levels.

Proposed method

• Propose a self-regularized luminance constraint as an unsupervised prior to stabilize illumination distribution in ultra-dark scenes.
• Design Illuminance-Semantic Dual Modulation (ISDM) as a refinement middleware that operates across decoding layers to jointly preserve illuminance and semantic color details.
• Implement ISDM with intermediate components: Illuminance Modulation Unit (IMU) and Semantic Modulation Unit (SMU), enabling cross-branch feature modulation.
• Introduce the Resolution-Sensitive Merging Up-sampler (RSMU) that fuses multiple sampling modalities as substrates, reducing artifacts and halos.
• Employ a multi-loss training strategy combining luminance, illumination, and perceptual losses to guide feature learning and improve realism.
• Train the end-to-end UltraBM framework on the RELLISUR dataset, optimizing for PSNR, SSIM, and LPIPS metrics.

Experimental results

Research questions

• RQ1 Can a unified framework effectively address both low-light enhancement and super-resolution in ultra-dark scenes, surpassing cascaded or single-task approaches?
• RQ2 How can illumination and semantic color priors be jointly modeled to preserve fine details and reduce color distortion in low-light super-resolution?
• RQ3 To what extent does a resolution-sensitive up-sampling strategy mitigate artifacts and halos compared to standard up-sampling methods?
• RQ4 How does the dual-stream architecture with ISDM improve feature-level preservation of luminance and texture details?
• RQ5 What is the generalization capability of the proposed method across varying levels of darkness, including extreme exposure conditions?

Key findings

• UltraBM achieves a 5% improvement in PSNR and a 43% reduction in LPIPS compared to state-of-the-art methods, demonstrating superior perceptual and quantitative performance.
• The method exhibits a 19-fold increase in RMSE score across different darkness levels, indicating exceptional generalization capability in ultra-low-light conditions.
• Ablation studies confirm that removing ISDM leads to a 0.553 dB PSNR drop, proving its critical role in preserving illumination and color details.
• The RSMU module improves PSNR by 0.404 dB over bilinear up-sampling, significantly reducing artifacts and halos.
• The perceptual loss ($\mathcal{L}_P$) has the most significant impact on LPIPS, highlighting its importance in enhancing visual realism.
• The self-regularized luminance constraint ($\mathcal{L}_{SL}$) contributes to a 0.31 dB PSNR gain and 0.014 LPIPS improvement, validating its effectiveness in stabilizing illumination.