[Paper Review] EarthPT: a time series foundation model for Earth Observation

Michael J. Smith, Luke Fleming+1 more|arXiv (Cornell University)|Sep 13, 2023

Remote-Sensing Image Classification4 citations

TL;DR

EarthPT is a 700 million parameter autoregressive transformer foundation model trained on 14 billion tokens of multispectral Earth Observation time series data, enabling accurate pixel-level forecasting of surface reflectance and NDVI up to five months in advance with a typical error of ~0.05. It also generates semantically meaningful embeddings useful for dynamic land use classification and other downstream tasks.

ABSTRACT

We introduce EarthPT -- an Earth Observation (EO) pretrained transformer. EarthPT is a 700 million parameter decoding transformer foundation model trained in an autoregressive self-supervised manner and developed specifically with EO use-cases in mind. We demonstrate that EarthPT is an effective forecaster that can accurately predict future pixel-level surface reflectances across the 400-2300 nm range well into the future. For example, forecasts of the evolution of the Normalised Difference Vegetation Index (NDVI) have a typical error of approximately 0.05 (over a natural range of -1 -> 1) at the pixel level over a five month test set horizon, out-performing simple phase-folded models based on historical averaging. We also demonstrate that embeddings learnt by EarthPT hold semantically meaningful information and could be exploited for downstream tasks such as highly granular, dynamic land use classification. Excitingly, we note that the abundance of EO data provides us with -- in theory -- quadrillions of training tokens. Therefore, if we assume that EarthPT follows neural scaling laws akin to those derived for Large Language Models (LLMs), there is currently no data-imposed limit to scaling EarthPT and other similar `Large Observation Models.'

Motivation & Objective

• To develop a self-supervised, autoregressive transformer foundation model tailored for Earth Observation (EO) time series data.
• To demonstrate that such a model can forecast surface reflectance and remote sensing indices (e.g., NDVI) months in advance with high accuracy.
• To show that EarthPT learns semantically meaningful representations that can be leveraged for downstream tasks like granular land use classification.
• To establish that EO data offers a vast, untapped data source for scaling Large Observation Models, with potential for quadrillions of training tokens.

Proposed method

• EarthPT is a decoding transformer model with 700 million parameters, trained in an autoregressive, self-supervised manner on multispectral time series from ClearSky-generated Sentinel-2-like reflectance data.
• Time series are constructed from 100×100 km BNG tiles, with 10 spectral bands (400–2300 nm) and date embeddings using sine and cosine encodings of day-of-year.
• Reflectance values are normalized to [-1, 1] via the transformation $\hat{v} = v/500 - 1$, and each temporal observation is treated as a single token.
• The model is trained on 14 billion tokens of time series data from January 2015 to December 2022 across a UK region, with inference tested on January–May 2023.
• Pixel-level embeddings are extracted from the penultimate transformer layer and averaged across time steps to form a 1280-dimensional representation.
• Dimensionality reduction via PCA is used to visualize and analyze the semantic structure of the learned embedding space.

Figure 1 : Predictions of some common remote sensing indicators for a randomly chosen pixel within the UK National Grid TL tile. We condition EarthPT on ClearSky time series from 1st January 2015 to 1st January 2023, with outputs after this divergence date constituting a long-term forecast to be com

Experimental results

Research questions

• RQ1 Can a self-supervised, autoregressive transformer model effectively forecast multispectral reflectance values in Earth Observation time series over long horizons (e.g., five months)?
• RQ2 Do the learned representations from EarthPT encode semantically meaningful information about land surface dynamics, such as vegetation cycles or surface properties?
• RQ3 Can EarthPT’s embeddings be used to support highly granular, dynamic land use classification tasks?
• RQ4 To what extent is EarthPT scalable given the vast volume of available Earth Observation data?

Key findings

• EarthPT achieves a typical forecasting error of approximately 0.05 in pixel-level NDVI predictions over a five-month horizon, outperforming simple phase-folded historical averaging models.
• The model successfully forecasts surface reflectance across the 400–2300 nm spectral range with high fidelity, enabling prediction of key remote sensing indices.
• PCA visualization of EarthPT embeddings reveals coherent, physically meaningful spatial organization, with distinct clustering by indices such as NDVI, BSI, and RGB.
• The learned embedding space shows strong potential for downstream tasks like dynamic land cover classification when combined with semantic labels.
• EarthPT is not currently data-limited, with access to over 4.3 trillion tokens in the UK alone and over a quadrillion tokens when scaling geographically, suggesting vast potential for future model scaling.
• Assuming Chinchilla scaling laws apply, EarthPT could be scaled to 50 trillion parameters—three orders of magnitude larger than current state-of-the-art LLMs—without data constraints.

Figure 2 : Median L1 error and interquartile ranges of NDVI predictions for 1M pixels in the TL63 tile. EarthPT long-term forecasts out-perform a simple phase-folded model based on historical averages out to a horizon of five months.