[Paper Review] DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data

Lixian Zhang, Runmin Dong+5 more|arXiv (Cornell University)|Feb 24, 2024

Impact of Light on Environment and HealthEnvironmental Science3 citations

TL;DR

This paper proposes DeepLightSR, a calibration-aware multi-modal super-resolution method that reconstructs high-resolution nighttime light (NTL) images using low-resolution NTL data fused with daytime multispectral, digital elevation, and impervious surface data. The approach achieves state-of-the-art performance with up to 13.25 dB PSNR improvement and 9.32 PIQE reduction over baselines, enabled by the novel DeepLightMD dataset and components like calibration-aware alignment and auxiliary-embedded refinement.

ABSTRACT

Nighttime light (NTL) remote sensing observation serves as a unique proxy for quantitatively assessing progress toward meeting a series of Sustainable Development Goals (SDGs), such as poverty estimation, urban sustainable development, and carbon emission. However, existing NTL observations often suffer from pervasive degradation and inconsistency, limiting their utility for computing the indicators defined by the SDGs. In this study, we propose a novel approach to reconstruct high-resolution NTL images using multi-modal remote sensing data. To support this research endeavor, we introduce DeepLightMD, a comprehensive dataset comprising data from five heterogeneous sensors, offering fine spatial resolution and rich spectral information at a national scale. Additionally, we present DeepLightSR, a calibration-aware method for building bridges between spatially heterogeneous modality data in the multi-modality super-resolution. DeepLightSR integrates calibration-aware alignment, an auxiliary-to-main multi-modality fusion, and an auxiliary-embedded refinement to effectively address spatial heterogeneity, fuse diversely representative features, and enhance performance in $8 imes$ super-resolution (SR) tasks. Extensive experiments demonstrate the superiority of DeepLightSR over 8 competing methods, as evidenced by improvements in PSNR (2.01 dB $ \sim $ 13.25 dB) and PIQE (0.49 $ \sim $ 9.32). Our findings underscore the practical significance of our proposed dataset and model in reconstructing high-resolution NTL data, supporting efficiently and quantitatively assessing the SDG progress.

Motivation & Objective

• To address the degradation and inconsistency in historical low-resolution nighttime light (NTL) observations, which limit their utility for Sustainable Development Goal (SDG) monitoring.
• To develop a method capable of reconstructing high-resolution NTL images from low-resolution inputs under complex, spatially heterogeneous degradations such as blurring, blooming, and sensor drift.
• To create a large-scale, nationally representative, multi-modal remote sensing dataset (DeepLightMD) integrating NTL, multispectral, DEM, and impervious surface data with geometric alignment and noise reduction.
• To design a deep learning framework (DeepLightSR) that effectively fuses multi-modal features while handling spatial misalignment and calibration inconsistencies.
• To enable more accurate, quantitative assessment of SDG progress—particularly in poverty estimation, urban development, and carbon emissions—through improved NTL data.

Proposed method

• DeepLightSR employs a calibration-aware alignment (CAA) module to implicitly align spatially heterogeneous modality features by calibrating the main modality and adjusting auxiliary inputs under complex degradations.
• An auxiliary-to-main multi-modality fusion (AMFF) module extracts and fuses low-level and high-level features from auxiliary modalities (DMO, DEM, ISP) in a stepwise, hierarchical manner to preserve representative and aligned features.
• An auxiliary-embedded refinement (AER) module uses on-the-fly supervision from the ISP product to guide detail reconstruction, enhancing perceptual quality and structural accuracy in 8× super-resolution.
• The model is trained with multi-scale supervisory signals and loss functions that include L1, perceptual, and structural similarity (SSIM) losses to improve realism and fidelity.
• The method integrates multi-modal inputs through a cross-attention mechanism that enables feature interaction across different modalities while maintaining spatial consistency.
• A comprehensive pre-processing pipeline is applied to DeepLightMD to correct geometrical errors, remove noise, and align all modalities spatially before model training.

Experimental results

Research questions

• RQ1 Can multi-modal remote sensing data improve the reconstruction quality of low-resolution nighttime light images beyond single-modality super-resolution?
• RQ2 How effectively can calibration-aware alignment resolve spatial misalignment and inconsistency across heterogeneous remote sensing modalities in NTL super-resolution?
• RQ3 What is the relative contribution of each auxiliary modality—daytime multispectral, digital elevation, and impervious surface data—to the final NTL reconstruction performance?
• RQ4 To what extent does auxiliary supervision from the impervious surface product enhance perceptual realism and structural detail recovery in high-resolution NTL images?
• RQ5 Can the proposed method generalize to real-world degradation patterns, including sensor blurring, atmospheric scattering, and over-saturation, better than existing simulated SR benchmarks?

Key findings

• DeepLightSR achieves a PSNR improvement of up to 13.25 dB and a PIQE reduction of 9.32 compared to baseline methods on the DeepLightMD dataset.
• Excluding the daytime multispectral (DMO) input results in a 4.4 dB drop in PSNR and a 2.46-point decrease in CC, indicating DMO's critical role in distinguishing human activity from natural areas.
• Removing the digital elevation model (DEM) leads to a 1.94-point drop in SAM and a 2.5-point decrease in CC, highlighting its importance in spatial calibration and alignment.
• The auxiliary-embedded refinement (AER) module contributes significantly to perceptual quality, reducing PIQE by 1.7 points compared to the ablated version without it.
• The full model with all components achieves a PSNR of 32.39 dB and a PIQE of 8.55, outperforming all 8 competing methods in all evaluation metrics.
• Ablation studies confirm that each component—CAA, AMFF, and AER—plays a distinct and essential role: CAA enables alignment, AMFF enhances feature fusion, and AER improves structural detail recovery and blooming suppression.