[Paper Review] Evaluation Challenges for Geospatial ML

Esther Rolf|arXiv (Cornell University)|Mar 31, 2023

Human Mobility and Location-Based Analysis15 citations

TL;DR

The paper surveys evaluation challenges unique to geospatial ML and maps, contrasting map accuracy with model performance, and offers concrete opportunities to improve evaluation practices.

ABSTRACT

As geospatial machine learning models and maps derived from their predictions are increasingly used for downstream analyses in science and policy, it is imperative to evaluate their accuracy and applicability. Geospatial machine learning has key distinctions from other learning paradigms, and as such, the correct way to measure performance of spatial machine learning outputs has been a topic of debate. In this paper, I delineate unique challenges of model evaluation for geospatial machine learning with global or remotely sensed datasets, culminating in concrete takeaways to improve evaluations of geospatial model performance.

Motivation & Objective

• Motivate the need for rigorous evaluation of geospatial ML predictions due to their downstream use in science and policy.
• Explain how geospatial data structure (spatial autocorrelation, covariate shift) complicates traditional ML evaluation.
• Differentiate between measuring map accuracy as a population parameter and assessing broader model performance.
• Propose concrete opportunities to improve evaluation data, frameworks, and transparency for geospatial ML.
• Highlight spatially-aware validation methods and their limitations in reflecting real-world performance.

Proposed method

• Define map accuracy as a population parameter with notation for predicted and ground-truth values across a target domain.
• Discuss design-based vs model-based estimation for accuracy using probability samples when available.
• Differentiate map accuracy from broader model performance including spatial generalization, interpretability, and usability.
• Review spatially-aware evaluation methods (spatial cross-validation, buffered validation, extrapolation-focused designs) and their effects on reported performance.
• Suggest complementary evaluation practices such as visualization of residuals and baseline comparisons to contextualize results.
• Outline three opportunities to improve the evaluation landscape: better evaluation data, evaluation frameworks, and transparent reporting of limitations.

Experimental results

Research questions

• RQ1 What are the distinct evaluation challenges posed by geospatial ML compared to traditional ML?
• RQ2 How do spatial structure and data gaps affect the validity of standard evaluation procedures?
• RQ3 What evaluation methods are appropriate for quantifying map accuracy versus broader model performance in geospatial contexts?
• RQ4 What concrete opportunities can improve the reliability and transparency of geospatial ML evaluations?

Key findings

• Map accuracy should be treated as a population parameter to be estimated, with design-based inference possible when evaluation data come from a design-independent probability sample.
• Performance in geospatial ML extends beyond map accuracy to include spatial generalization, extrapolation, interpretability, and usability.
• Spatial correlation and covariate shift can inflate non-spatial validation results and affect generalization assessments; thus, non-spatial splits are often unsuitable.
• Spatial cross-validation reduces spatial dependence but can under-report performance in interpolation regimes, highlighting the need for diverse validation designs.
• Three opportunities are proposed: invest in evaluation data, invest in evaluation frameworks, and clearly communicate limitations when data are insufficient.