[Paper Review] Seeing Through Clouds in Satellite Images

M. Zhao, Peder A. Olsen+1 more|arXiv (Cornell University)|Jun 15, 2021

Advanced Image Fusion Techniques4 citations

TL;DR

This paper proposes SpaceEye, a novel neural network model that reconstructs cloud-free multispectral satellite images by fusing multi-temporal optical satellite data with radio frequency (RF) signals from Synthetic Aperture Radar (SAR), enabling accurate, daily cloud removal. The method achieves a significant 8dB improvement in PSNR over baselines and enables critical applications in agriculture, flood monitoring, and wildfire detection using publicly available data.

ABSTRACT

This paper presents a neural-network-based solution to recover pixels occluded by clouds in satellite images. We leverage radio frequency (RF) signals in the ultra/super-high frequency band that penetrate clouds to help reconstruct the occluded regions in multispectral images. We introduce the first multi-modal multi-temporal cloud removal model. Our model uses publicly available satellite observations and produces daily cloud-free images. Experimental results show that our system significantly outperforms baselines by 8dB in PSNR. We also demonstrate use cases of our system in digital agriculture, flood monitoring, and wildfire detection. We will release the processed dataset to facilitate future research.

Motivation & Objective

• To address the persistent challenge of cloud occlusion in optical satellite imagery, which limits timely monitoring of environmental changes.
• To develop a method that reconstructs accurate, cloud-free multispectral images using complementary RF signals that penetrate clouds.
• To create a multi-modal, multi-temporal deep learning framework that fuses optical and SAR data without requiring ground-truth cloud-free images.
• To enable practical applications in digital agriculture, flood monitoring, and wildfire detection by providing consistent, daily cloud-free image outputs.
• To release a processed dataset to support future research in cloud-removal and multi-modal Earth observation.

Proposed method

• The model uses a multi-modal attention mechanism to fuse multispectral images and SAR-derived RF reflectivity maps, aligning them spatially and temporally.
• It introduces a modified adversarial training scheme that operates with partial observations of the target distribution, enabling high-fidelity generation without ground-truth cloud-free images.
• The system leverages publicly available Sentinel-1 (SAR) and Sentinel-2 (multispectral) data, processed to align temporal and spatial dimensions.
• A novel optimization framework combines damped interpolation and low-rank matrix completion to reconstruct missing pixels, with a fixed rank of 35 for computational efficiency.
• The method employs a closed-form iterative algorithm using pre-computed inverse matrices for GPU-accelerated inference, minimizing computational overhead.
• The model is trained end-to-end using a combination of perceptual and adversarial losses to preserve structural and spectral fidelity in reconstructed images.

Experimental results

Research questions

• RQ1 Can RF signals from SAR be effectively used to reconstruct cloud-occluded optical satellite imagery with high accuracy?
• RQ2 How can multi-modal (optical and RF) and multi-temporal satellite data be fused to enable daily cloud-free image prediction?
• RQ3 Can a deep learning model achieve high-fidelity cloud removal without ground-truth cloud-free images?
• RQ4 What is the performance gain of multi-modal, multi-temporal fusion over existing single-modality or non-temporal approaches in cloud removal?
• RQ5 To what extent can the proposed method improve monitoring in time-sensitive applications like flood tracking and crop monitoring?

Key findings

• The proposed SpaceEye model achieves an 8dB improvement in PSNR over existing baselines, significantly outperforming state-of-the-art methods in cloud removal.
• The system successfully reconstructs the extent of standing water during a flood event in Carnation, WA, accurately capturing flood dynamics on February 7th and drainage after mid-February despite persistent cloud cover.
• The model demonstrates robust performance in digital agriculture by enabling consistent monitoring of crop emergence and growth cycles.
• The system enables reliable wildfire detection by providing cloud-free imagery for early detection of burn scars and post-fire recovery.
• The method achieves high-fidelity reconstruction even without ground-truth cloud-free images, thanks to a novel adversarial training scheme using partial observations.
• The researchers will release a processed dataset derived from publicly available Sentinel missions to support future research in multi-modal Earth observation.