[Paper Review] DiffusionSat: A Generative Foundation Model for Satellite Imagery

Samar Khanna, Patrick Liu+6 more|arXiv (Cornell University)|Dec 6, 2023

Computational and Text Analysis MethodsSocial Sciences31 citations

TL;DR

DiffusionSat is the first large-scale latent-diffusion generative model for satellite imagery, conditioning on text and metadata to enable single-image generation and 3D control for super-resolution, temporal generation, and in-painting.

ABSTRACT

Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregularly sampled across time -- and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently spatio-temporal, requiring conditional generation tasks not supported by traditional methods based on captions or images. In this paper, we present DiffusionSat, to date the largest generative foundation model trained on a collection of publicly available large, high-resolution remote sensing datasets. As text-based captions are sparsely available for satellite images, we incorporate the associated metadata such as geolocation as conditioning information. Our method produces realistic samples and can be used to solve multiple generative tasks including temporal generation, superresolution given multi-spectral inputs and in-painting. Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale generative foundation model for satellite imagery. The project website can be found here: https://samar-khanna.github.io/DiffusionSat/

Motivation & Objective

• Motivate the need for a diffusion-based generative model tailored to satellite imagery with its spectral, temporal, and metadata characteristics.
• Propose DiffusionSat, a Latent Diffusion Model trained on publicly available high-resolution satellite datasets, leveraging metadata as conditioning signals.
• Develop a 3D conditioning extension (ControlNet-like) to enable tasks such as multi-spectral super-resolution, temporal generation, and in-painting.
• Demonstrate that DiffusionSat achieves state-of-the-art results on satellite image generation and related inverse problems.
• Provide a public pre-training dataset and a training protocol that can adapt to diverse geospatial tasks.

Proposed method

• Use a latent diffusion framework (downsampling via VAE, diffusion in latent space, upsampling via decoder) initialized from Stable Diffusion weights.
• Encode numerical satellite metadata with sinusoidal projections and per-matum MLPs, summing into a final conditioning vector with the timestep embedding.
• Condition the denoising network on CLIP-like text captions when captions exist; otherwise rely on metadata and timestep conditioning.
• Introduce a 3D-ControlNet inspired conditioning mechanism to handle sequences of images for temporal generation, including temporal attention and 3D zero-convolutions between SD blocks.
• Train on multi-dataset satellite imagery (fMoW, Satlas, SpaceNet) with associated metadata (lat/long, timestamp, GSD, cloud cover, etc.).
• Apply the model to single-image generation and downstream conditional tasks: super-resolution, temporal prediction, and in-painting, demonstrating improved metrics over baselines.

Figure 1: Conditioning on freely available metadata and using large, publicly available satellite imagery datasets shows DiffusionSat is a powerful generative foundation model for remote sensing data.

Experimental results

Research questions

• RQ1 Can a diffusion-based foundation model be effectively trained on satellite imagery using metadata as conditioning signals to achieve high-quality single-image generation?
• RQ2 Can a 3D conditioning framework enable reliable multi-task generation on remote sensing data, including super-resolution, temporal forecasting, and in-painting?
• RQ3 What is the impact of metadata-aware conditioning versus text-only conditioning on generation quality and control in satellite imagery?
• RQ4 Do pre-trained latent-diffusion weights adapted to satellite data enable better downstream performance than models trained from scratch for inverse problems?
• RQ5 How well does the model generalize across diverse satellite datasets (fMoW, Satlas, SpaceNet) with varying GSD and spectral bands?

Key findings

• DiffusionSat achieves strong visual and perceptual quality for single-image satellite generation, outperforming baselines in FID, IS, and CLIP scores.
• Incorporating numerical metadata via sinusoidal embeddings and per-matum MLPs improves generation quality over caption-only conditioning.
• The 3D conditioning approach enables state-of-the-art or competitive performance on downstream tasks including multi-spectral super-resolution, temporal generation, and in-painting.
• DiffusionSat demonstrates better LPIPS and competitive SSIM/PSNR on temporal prediction and in-painting benchmarks compared to baselines like STSR and MCVD, across multiple datasets.
• Pretraining on large, public satellite datasets and freezing most Stable Diffusion weights while training only the denoising net and metadata encoders speeds convergence and leverages existing weights.

Figure 2: DiffusionSat flexibly extends to a variety of conditional generation tasks. We design a 3D version of a ControlNet (Zhang & Agrawala, 2023 ) which can accept a sequence of images. Like regular ControlNets, our 3D ControlNet keeps a trainable copy of SD weights for the downsampling and midd