[Paper Review] Uncertainty quantification for probabilistic machine learning in earth observation using conformal prediction

Geethen Singh, Glenn R. Moncrieff+4 more|arXiv (Cornell University)|Jan 12, 2024

Atmospheric and Environmental Gas DynamicsEnvironmental Science3 citations

TL;DR

This paper introduces a model-agnostic uncertainty quantification framework using conformal prediction for probabilistic machine learning in Earth Observation, enabling statistically valid, computationally efficient prediction regions without access to training data. It demonstrates seamless integration into Google Earth Engine workflows across local-to-global datasets, improving reliability in decision-critical EO applications like land cover mapping and biomass estimation.

ABSTRACT

Unreliable predictions can occur when using artificial intelligence (AI) systems with negative consequences for downstream applications, particularly when employed for decision-making. Conformal prediction provides a model-agnostic framework for uncertainty quantification that can be applied to any dataset, irrespective of its distribution, post hoc. In contrast to other pixel-level uncertainty quantification methods, conformal prediction operates without requiring access to the underlying model and training dataset, concurrently offering statistically valid and informative prediction regions, all while maintaining computational efficiency. In response to the increased need to report uncertainty alongside point predictions, we bring attention to the promise of conformal prediction within the domain of Earth Observation (EO) applications. To accomplish this, we assess the current state of uncertainty quantification in the EO domain and found that only 20% of the reviewed Google Earth Engine (GEE) datasets incorporated a degree of uncertainty information, with unreliable methods prevalent. Next, we introduce modules that seamlessly integrate into existing GEE predictive modelling workflows and demonstrate the application of these tools for datasets spanning local to global scales, including the Dynamic World and Global Ecosystem Dynamics Investigation (GEDI) datasets. These case studies encompass regression and classification tasks, featuring both traditional and deep learning-based workflows. Subsequently, we discuss the opportunities arising from the use of conformal prediction in EO. We anticipate that the increased availability of easy-to-use implementations of conformal predictors, such as those provided here, will drive wider adoption of rigorous uncertainty quantification in EO, thereby enhancing the reliability of uses such as operational monitoring and decision making.

Motivation & Objective

• To address the lack of reliable uncertainty reporting in Earth Observation (EO) machine learning applications.
• To introduce a model-agnostic uncertainty quantification method compatible with diverse EO datasets and models.
• To enable statistically valid, computationally efficient prediction regions without requiring access to training data or model architecture.
• To demonstrate the practical integration of conformal prediction into existing Google Earth Engine workflows for regression and classification tasks.
• To promote wider adoption of rigorous uncertainty reporting in operational EO monitoring and decision-making systems.

Proposed method

• The paper applies conformal prediction as a post-hoc uncertainty quantification framework, operating independently of the underlying model and training data.
• It uses nonconformity scores computed on a calibration set to define prediction regions with guaranteed coverage probability.
• The method is adapted for both regression and classification tasks, with separate calibration procedures for each.
• The framework is implemented as modular tools compatible with Google Earth Engine, enabling integration into existing ML pipelines.
• It supports both traditional and deep learning models, maintaining computational efficiency across local and global EO datasets.
• The approach ensures marginal and conditional coverage guarantees under i.i.d. assumptions, providing statistically valid uncertainty estimates.

Experimental results

Research questions

• RQ1 Can conformal prediction provide statistically valid uncertainty quantification for Earth Observation models without access to training data or model structure?
• RQ2 How well does conformal prediction perform in terms of coverage and efficiency across diverse EO datasets, including Dynamic World and GEDI?
• RQ3 To what extent can conformal prediction be integrated into existing Google Earth Engine-based ML workflows for operational use?
• RQ4 How does conformal prediction compare to existing uncertainty quantification methods in EO, particularly in terms of reliability and computational cost?
• RQ5 What are the practical implications of adopting conformal prediction for decision-making in Earth observation applications?

Key findings

• Only 20% of reviewed Google Earth Engine datasets included any form of uncertainty information, highlighting a critical gap in current EO practice.
• The proposed conformal prediction framework achieved valid coverage rates across both regression and classification tasks, meeting theoretical guarantees.
• The method maintained computational efficiency, enabling scalable deployment on global-scale EO datasets such as Dynamic World and GEDI.
• The integration of conformal prediction modules into Google Earth Engine workflows was seamless, supporting both traditional and deep learning models.
• The approach outperformed commonly used but unreliable uncertainty estimation techniques prevalent in the EO literature.
• The study demonstrates that conformal prediction can significantly enhance the reliability of ML predictions in operational Earth observation applications.