[Paper Review] Learning Disentangled Representations of Satellite Image Time Series

Eduardo Hugo Sanchez, Mathieu Serrurier+1 more|arXiv (Cornell University)|Mar 21, 2019

Time Series Analysis and Forecasting21 references3 citations

TL;DR

This paper proposes a novel VAE-GAN hybrid model with cross-domain autoencoders to learn disentangled representations from satellite image time series, separating shared (common) and exclusive (image-specific) features. Trained on Sentinel-2 data, the model enables state-of-the-art performance in downstream tasks like classification, segmentation, and change detection using only unsupervised pretraining.

ABSTRACT

In this paper, we investigate how to learn a suitable representation of satellite image time series in an unsupervised manner by leveraging large amounts of unlabeled data. Additionally , we aim to disentangle the representation of time series into two representations: a shared representation that captures the common information between the images of a time series and an exclusive representation that contains the specific information of each image of the time series. To address these issues, we propose a model that combines a novel component called cross-domain autoencoders with the variational autoencoder (VAE) and generative ad-versarial network (GAN) methods. In order to learn disentangled representations of time series, our model learns the multimodal image-to-image translation task. We train our model using satellite image time series from the Sentinel-2 mission. Several experiments are carried out to evaluate the obtained representations. We show that these disentangled representations can be very useful to perform multiple tasks such as image classification, image retrieval, image segmentation and change detection.

Motivation & Objective

• To learn unsupervised, disentangled representations from large volumes of unlabeled satellite image time series.
• To separate a time series into a shared representation (common across images) and exclusive representations (specific to each image).
• To enable downstream tasks such as image classification, retrieval, segmentation, and change detection using only unsupervised features.
• To demonstrate the effectiveness of disentangled features on real-world satellite data, particularly Sentinel-2 time series.

Proposed method

• The model combines variational autoencoders (VAE) and generative adversarial networks (GANs) to learn a low-dimensional latent space with disentangled factors.
• A cross-domain autoencoder component is introduced to enforce image-to-image translation between time series images, enabling disentanglement of shared and exclusive features.
• The architecture supports multimodal generation, allowing one input image to generate multiple plausible outputs corresponding to different time points.
• Shared features are extracted via a shared encoder, while exclusive features are learned through a dedicated encoder per image, enabling disentanglement.
• The model is trained end-to-end using reconstruction loss, adversarial loss, and KL divergence regularization to ensure disentanglement and distributional consistency.
• The framework is applied to Sentinel-2 satellite image time series, handling high-dimensional, large-scale data effectively.

Experimental results

Research questions

• RQ1 Can a unified deep generative model learn disentangled representations from satellite image time series in an unsupervised manner?
• RQ2 Can the shared representation capture common spatial and textural patterns across time series images, while exclusive representations encode time-specific changes?
• RQ3 Does the disentangled representation improve performance on downstream tasks such as image classification and change detection?
• RQ4 Can the model generalize across diverse satellite image time series with varying temporal dynamics?

Key findings

• The model achieves 92.38% accuracy on the EuroSAT dataset using only unsupervised pretraining with a frozen feature extractor, significantly outperforming random initialization (62.13%).
• Fine-tuning the pretrained feature extractor improves accuracy to 94.54% within 10 epochs, demonstrating strong transferability of the learned features.
• Unsupervised image segmentation using shared features achieves visually coherent results, correctly identifying rivers, ports, and residential areas in Shanghai.
• The exclusive feature representation enables a simple yet effective change detection method using L1 distance between features, producing visually coherent change maps.
• The disentangled representation enables consistent performance across multiple tasks, including classification, retrieval, segmentation, and change detection, without any labeled data.
• The model successfully processes high-dimensional Sentinel-2 time series, demonstrating scalability and robustness to large-scale satellite data.