[Paper Review] Conditional Denoising of Remote Sensing Imagery Using Cycle-Consistent Deep Generative Models

M. Yu. Zotov, Jevgenij Gamper|arXiv (Cornell University)|Oct 31, 2019

Image and Signal Denoising Methods4 citations

TL;DR

This paper proposes a cycle-consistent conditional generative model to denoise cloudy Sentinel-2 remote sensing images using cloud-penetrating spectral bands. By leveraging adversarial training and Fréchet Distance-based evaluation in classifier feature space, the method achieves state-of-the-art land cover classification performance and demonstrates effective inpainting of cloud and shadow regions, though mode collapse remains a challenge in practice.

ABSTRACT

The potential of using remote sensing imagery for environmental modelling and for providing real time support to humanitarian operations such as hurricane relief efforts is well established. These applications are substantially affected by missing data due to non-structural noise such as clouds, shadows and other atmospheric effects. In this work we probe the potential of applying a cycle-consistent latent variable deep generative model (DGM) for denoising cloudy Sentinel-2 observations conditioned on the information in cloud penetrating bands. We adapt the recently proposed Fr\\'{e}chet Distance metric to remote sensing images for evaluating performance of the generator, demonstrate the potential of DGMs for conditional denoising, and discuss future directions as well as the limitations of DGMs in Earth science and humanitarian applications.

Motivation & Objective

• To address the critical problem of cloud and shadow-induced data loss in high-resolution remote sensing imagery, which hinders environmental monitoring and humanitarian applications.
• To develop a conditional generative model that leverages cloud-penetrating bands to reconstruct clear-sky surface reflectance from cloudy observations.
• To evaluate generative model fidelity using a novel Fréchet Distance-based metric in the feature space of a trained image classifier, enabling robust assessment of generated image quality.
• To demonstrate state-of-the-art performance in land cover classification on the BigEarthNet dataset using denoised imagery.
• To identify limitations of deep generative models in Earth science, particularly mode collapse and distribution shift, and propose directions for future improvement.

Proposed method

• The method employs an augmented cycle-consistent GAN (Aug CycleGAN) with two generator-discriminator pairs: one mapping cloudy images to clear images, and another mapping clear images back to cloudy ones, enforcing cycle consistency.
• Latent variables are introduced to model cloud and shadow positions, with the generator conditioned on observed cloud-penetrating bands to guide reconstruction.
• Spectral normalization is applied to convolutional layers to stabilize training and improve GAN convergence.
• The model is trained on paired mini-batches of cloudy and clear images from the BigEarthNet dataset, with cycle-consistency enforced via L1 loss on reconstructed images.
• A pre-trained 18-layer ResNet is used as a feature extractor to compute Fréchet Distance between real and generated image distributions in the classifier’s feature space.
• Model evaluation includes both Fréchet Distance and Inception Score, with the latter used to detect mode collapse when FD is misleading.

Experimental results

Research questions

• RQ1 Can cycle-consistent conditional generative models effectively reconstruct clear-sky surface reflectance from cloudy Sentinel-2 images using cloud-penetrating spectral bands?
• RQ2 How well can the Fréchet Distance metric in the classifier feature space assess the fidelity of generated remote sensing images compared to human perception or downstream task performance?
• RQ3 To what extent does mode collapse affect the quality of denoised remote sensing images, and can alternative metrics like Inception Score better capture this issue in multi-label settings?
• RQ4 How does the proposed method compare to existing cloud masking or denoising techniques in terms of downstream land cover classification accuracy?
• RQ5 What are the key limitations of deep generative models in Earth observation applications, particularly regarding distributional shifts and class imbalance?

Key findings

• The proposed model achieves state-of-the-art performance in land cover classification on the BigEarthNet dataset when using denoised images, demonstrating the utility of the approach for downstream tasks.
• The Fréchet Distance metric effectively captures distributional similarity between real and generated images in the classifier feature space, providing a reliable evaluation benchmark.
• Despite low Fréchet Distance scores, some models exhibited mode collapse, producing light-brown inpainted regions in place of clouds, indicating that FD alone is insufficient for assessing sample diversity.
• The Inception Score, which measures class diversity, was found to be a better indicator of mode collapse than Fréchet Distance in this context, suggesting the need for multi-metric evaluation.
• The model successfully reconstructs land cover features under clouds and shadows (e.g., in images [1,3], [1,1], [3,3]), though struggles with thick cloud cover, as seen in [2,1] and [1,2].
• Model failure cases, such as incorrect translation in image [3,2], highlight the challenges in generalization and the need for improved architectural design and training strategies.