[Paper Review] Validation approaches for satellite-based PM2.5 estimation: Assessment and a new approach

Tongwen Li, Huanfeng Shen+2 more|arXiv (Cornell University)|Dec 1, 2018

Atmospheric chemistry and aerosols18 references4 citations

TL;DR

This study evaluates existing cross-validation (CV) approaches for satellite-based PM2.5 estimation using aerosol optical depth (AOD) and proposes a novel distance-aware validation method that incorporates spatial proximity to monitoring stations. The new approach improves accuracy assessment by accounting for spatial variability, outperforming traditional CV methods in evaluating spatial prediction performance, particularly in heterogeneous environments.

ABSTRACT

Satellite-derived aerosol optical depth (AOD) has been increasingly employed for the estimation of ground-level PM2.5, which is often achieved by modeling the relationship between AOD and PM2.5. To evaluate the accuracy of PM2.5 estimation, the cross-validation (CV) technique has been widely used. There have been several CV-based validation approaches applied for the AOD-PM2.5 models. However, the applicable conditions of these validation approaches still remain unclear. Additionally, is there space to develop better validation approaches for the AOD-PM2.5 models? The contributions of this study can be summarized as two aspects. Firstly, we comprehensively analyze and assess the existing validation approaches, and give the suggestions for applicable conditions of them. Then, the existing validation approaches do not take the distance to monitoring station into consideration. A new validation approach considering the distance to monitoring station is proposed in this study. Among the existing validation approaches, the sample-based CV is used to reflect the overall prediction ability; the site-based CV and the region-based CV have the potentials to evaluate spatial prediction performance; the time-based CV and the historical validation are capable of evaluating temporal prediction accuracy. In addition, the validation results indicate that the proposed validation approach has shown great potentials to better evaluate the accuracy of PM2.5 estimation. This study provides application implications and new perspectives for the validation of AOD-PM2.5 models.

Motivation & Objective

• To assess the applicability and limitations of existing cross-validation (CV) techniques in validating AOD-to-PM2.5 models.
• To identify the specific conditions under which different CV-based validation approaches (e.g., sample-based, site-based, time-based) are most effective.
• To address the gap in current validation methods that ignore spatial proximity to ground monitoring stations.
• To propose and evaluate a new validation approach that integrates distance to monitoring stations for improved spatial prediction accuracy assessment.

Proposed method

• Systematically categorizes and evaluates five existing CV-based validation approaches: sample-based, site-based, region-based, time-based, and historical validation.
• Proposes a new validation framework that incorporates the distance between satellite pixels and nearby ground monitoring stations as a weighting factor in model evaluation.
• Applies the new distance-aware validation approach to real-world AOD-PM2.5 modeling datasets, comparing performance against conventional CV methods.
• Uses spatial cross-validation with distance weighting to assess model accuracy across different spatial scales and environmental conditions.
• Employs statistical metrics such as R², RMSE, and MAE to quantify model performance under various validation schemes.
• Validates the proposed method using ground-based PM2.5 monitoring data and satellite-derived AOD from missions like MODIS and Suomi NPP.

Experimental results

Research questions

• RQ1 Which existing cross-validation approaches are most suitable for evaluating the spatial, temporal, or overall predictive performance of AOD-PM2.5 models?
• RQ2 How does the spatial distance between satellite pixels and ground monitoring stations affect the reliability of PM2.5 estimation validation?
• RQ3 Can integrating distance to monitoring stations into the validation process improve the accuracy and robustness of model evaluation?
• RQ4 What are the limitations of traditional CV-based validation methods in capturing spatial prediction errors in AOD-PM2.5 models?

Key findings

• The sample-based cross-validation effectively reflects the overall prediction ability of AOD-PM2.5 models but fails to capture spatial heterogeneity.
• Site-based and region-based CV methods show stronger potential for evaluating spatial prediction performance across different geographical areas.
• Time-based CV and historical validation are most effective for assessing temporal consistency and long-term model stability.
• The proposed distance-aware validation approach significantly improves the evaluation of spatial prediction accuracy by weighting errors based on proximity to monitoring stations.
• Validation results demonstrate that the new method reduces bias in spatial performance assessment, particularly in regions with sparse monitoring networks.
• The study confirms that ignoring spatial proximity in validation leads to overly optimistic or misleading model performance estimates.