[Paper Review] Assessing the performance of spatial cross-validation approaches for models of spatially structured data

M. J. Mahoney, Lucas K. Johnson+4 more|arXiv (Cornell University)|Mar 13, 2023

demographic modeling and climate adaptation10 citations

TL;DR

The paper compares spatial cross-validation methods for spatially structured data via landscape simulations, identifying that spatial CV with spatial out blocks and buffers yields more accurate performance estimates, and introduces the spatialsample R package within the tidymodels framework.

ABSTRACT

Evaluating models fit to data with internal spatial structure requires specific cross-validation (CV) approaches, because randomly selecting assessment data may produce assessment sets that are not truly independent of data used to train the model. Many spatial CV methodologies have been proposed to address this by forcing models to extrapolate spatially when predicting the assessment set. However, to date there exists little guidance on which methods yield the most accurate estimates of model performance. We conducted simulations to compare model performance estimates produced by five common CV methods fit to spatially structured data. We found spatial CV approaches generally improved upon resubstitution and V-fold CV estimates, particularly when approaches which combined assessment sets of spatially conjunct observations with spatial exclusion buffers. To facilitate use of these techniques, we introduce the `spatialsample` package which provides tooling for performing spatial CV as part of the broader tidymodels modeling framework.

Motivation & Objective

• Motivate the need for spatial cross-validation to avoid over-optimistic performance due to spatial autocorrelation in training data.
• Systematically compare leading spatial CV approaches using simulated landscapes and random forests.
• Identify parameter settings that yield accurate model performance estimates and provide practical guidance.
• Offer an accessible software tool to implement spatial CV within the tidymodels ecosystem.

Proposed method

• Simulated 100 independent landscapes on a 50x50 grid with 13 predictors and a derived target variable y.
• Compared multiple spatial CV approaches (blocked CV, clustered CV, BLO3 CV, LODO CV, BLO3 CV, V-fold CV) against resubstitution and standard randomized CV.
• Employed random forests (ranger) using predictors X2, X3, X6–X10 to predict y.
• Measured model performance with RMSE and compared estimates to a ‘true’ RMSE range derived from cross-landscape predictions.
• Calculated spatial autocorrelation ranges for y and residuals to inform appropriate D_in/D_out spacing.
• Implemented all methods in spatialsample, aligned with tidymodels infrastructure.

Figure 1: One simulation of the dependent variable $y$ and several of the CV approaches used. A: Simulated data showing environmental clustering of a variable of interest, named “Target”, in one simulated landscape. B: Spatial clustering CV fold assignments, as produced by spatial_clustering_cv() ,

Experimental results

Research questions

• RQ1 How do different spatial cross-validation approaches perform in estimating model generalization for spatially structured data?
• RQ2 Do spatial CV methods that combine spatially conjunct assessment data with exclusion buffers provide more accurate performance estimates than other approaches?
• RQ3 Which parameterizations (block size, number of folds, buffers, inclusion radii) yield RMSE estimates closest to the true RMSE range?
• RQ4 How does spacing between D_in and D_out influence the bias of performance estimates?
• RQ5 Can practical tools within tidymodels effectively implement these spatial CV methods for applied modeling pipelines?

Key findings

• Spatial CV methods generally yield more accurate performance estimates than resubstitution or randomized CV.
• Methods that combine D_out of spatially conjunct observations with exclusion buffers provided the best estimates of model performance.
• Spatial clustering and leave-one-disc-out (LODO) CV were among the most consistently effective approaches across parameterizations.
• Too aggressive data exclusion (e.g., small numbers of folds or overly large blocks) produced pessimistic RMSE estimates.
• The best-performing parameter sets separated D_out from D_in by roughly 25–41% of the grid length, aligning with the outcome’s autocorrelation range.
• Clustered CV showed robustness across parameter settings but may reflect a narrower parameter space examined.

Figure 2: RMSE distributions of each spatial CV method evaluated. The x axis represents the distance of an RMSE from the mean “true” RMSE values, such that negative values underpredict the true RMSE and positive values overpredict it. Distributions are scaled relative to the number of iterations run