[Paper Review] Generating Physically-Consistent Satellite Imagery for Climate Visualizations

Björn Lütjens, Brandon Leshchinskiy+12 more|arXiv (Cornell University)|Apr 10, 2021

Flood Risk Assessment and Management35 citations

TL;DR

This paper introduces a physics-informed GAN pipeline (Earth Intelligence Engine) that generates photorealistic satellite imagery of future coastal floods and ensures physical-consistency with a validated flood model (SLOSH). It also provides a new evaluation metric FVPS and an open dataset of image triplets for Earth observation image-to-image translation.

ABSTRACT

Deep generative vision models are now able to synthesize realistic-looking satellite imagery. But, the possibility of hallucinations prevents their adoption for risk-sensitive applications, such as generating materials for communicating climate change. To demonstrate this issue, we train a generative adversarial network (pix2pixHD) to create synthetic satellite imagery of future flooding and reforestation events. We find that a pure deep learning-based model can generate photorealistic flood visualizations but hallucinates floods at locations that were not susceptible to flooding. To address this issue, we propose to condition and evaluate generative vision models on segmentation maps of physics-based flood models. We show that our physics-conditioned model outperforms the pure deep learning-based model and a handcrafted baseline. We evaluate the generalization capability of our method to different remote sensing data and different climate-related events (reforestation). We publish our code and dataset which includes the data for a third case study of melting Arctic sea ice and $>$30,000 labeled HD image triplets -- or the equivalent of 5.5 million images at 128x128 pixels -- for segmentation guided image-to-image translation in Earth observation. Code and data is available at \url{https://github.com/blutjens/eie-earth-public}.

Motivation & Objective

• Motivate better climate impact communication by visualizing coastal floods as satellite imagery rather than color-coded maps or street views.
• Integrate physics-based flood model outputs with a deep learning image synthesis model to ensure physically-consistent imagery.
• Develop a quantitative metric to jointly evaluate photorealism and physical-consistency of generated imagery.
• Demonstrate generalization to other climate impacts (e.g., reforestation) and publish a large open dataset for image-to-image translation in Earth observation.

Proposed method

• Extend pix2pixHD to take a pre-flood image and a binary flood extent map as input at 1024×1024 resolution.
• Train the model to produce post-flood satellite imagery that is photorealistic and physically-consistent with the flood extent map.
• Define physical-consistency via IoU between water in the generated image and the flood map, using a flood segmentation model to compute water masks.
• Propose Flood Visualization Plausibility Score (FVPS) as the harmonic mean of IoU and (1 - LPIPS) to jointly assess physical-consistency and photorealism.
• Use a physics-based flood model (SLOSH) to generate binary flood maps and ground-truth-like inputs for training and evaluation.
• Publish an open dataset with over 25k labeled image-triplets to support image-to-image translation research in Earth observation.

Experimental results

Research questions

• RQ1 Can a GAN generate post-flood satellite imagery that is both photorealistic and physically-consistent with an expert-validated flood model?
• RQ2 Does incorporating physics-based flood maps into the conditioning input improve both physical-consistency and realism compared to baselines?
• RQ3 How does the proposed FVPS metric reflect improvements in both fidelity and physical alignment?
• RQ4 Can the approach generalize to visualize other climate impacts such as reforestation?

Key findings

• The physics-informed GAN outperforms a baseline GAN without physics in IoU, LPIPS, and FVPS on post-flood imagery.
• A handcrafted baseline can achieve physical consistency but lacks photorealism, showing the trade-offs in typical visualization tools.
• The learned model achieves IoU = 0.553, LPIPS = 0.263, and FVPS = 0.532 on high-resolution evaluation, outperforming baselines.
• Compared to a physics-informed VAEGAN, the GAN-based approach yields better physical-consistency and photorealism in most settings.
• The dataset and model generalize to reforestation visualization, with quantified improvements over color-masked baselines.
• Generalization across instruments and locations is feasible, demonstrated through supplementary experiments and a 6500-image-triplet im2im translation dataset.