[Paper Review] SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

Christopher Yeh, Chenlin Meng+8 more|arXiv (Cornell University)|Nov 8, 2021

Sustainability and Climate Change GovernanceEnvironmental Science21 citations

TL;DR

SustainBench introduces a standardized benchmark suite of 15 machine learning tasks across 7 Sustainable Development Goals (SDGs), using publicly released datasets for 11 tasks. It enables consistent evaluation, lowers entry barriers for researchers, and promotes novel methods like meta-learning and self-supervised learning to improve SDG monitoring using satellite and survey data.

ABSTRACT

Progress toward the United Nations Sustainable Development Goals (SDGs) has been hindered by a lack of data on key environmental and socioeconomic indicators, which historically have come from ground surveys with sparse temporal and spatial coverage. Recent advances in machine learning have made it possible to utilize abundant, frequently-updated, and globally available data, such as from satellites or social media, to provide insights into progress toward SDGs. Despite promising early results, approaches to using such data for SDG measurement thus far have largely evaluated on different datasets or used inconsistent evaluation metrics, making it hard to understand whether performance is improving and where additional research would be most fruitful. Furthermore, processing satellite and ground survey data requires domain knowledge that many in the machine learning community lack. In this paper, we introduce SustainBench, a collection of 15 benchmark tasks across 7 SDGs, including tasks related to economic development, agriculture, health, education, water and sanitation, climate action, and life on land. Datasets for 11 of the 15 tasks are released publicly for the first time. Our goals for SustainBench are to (1) lower the barriers to entry for the machine learning community to contribute to measuring and achieving the SDGs; (2) provide standard benchmarks for evaluating machine learning models on tasks across a variety of SDGs; and (3) encourage the development of novel machine learning methods where improved model performance facilitates progress towards the SDGs.

Motivation & Objective

• Address the lack of standardized evaluation and data access in machine learning for Sustainable Development Goal (SDG) monitoring.
• Lower barriers to entry for machine learning researchers by providing high-quality, domain-specific datasets from development economics and environmental science.
• Establish a common benchmark framework to track progress and compare model performance across diverse SDG-related tasks.
• Encourage the development of novel machine learning methods—such as meta-learning and self-supervised learning—tailored to real-world SDG challenges.
• Ensure ethical deployment by addressing data privacy and bias risks in geospatial and survey-based datasets.

Proposed method

• Curated 15 benchmark tasks spanning 7 SDGs, including poverty prediction, land cover classification, crop yield estimation, and water sanitation monitoring.
• Released 11 of the 15 datasets publicly for the first time, with standardized train/val/test splits and consistent evaluation protocols.
• Provided baseline models using state-of-the-art techniques, including MAML for meta-learning on land cover classification.
• Employed diverse data sources: satellite imagery (Landsat, Sentinel-1/2, MODIS), social media, mobile phone records, and ground-truth survey data (DHS, LSMS).
• Applied data augmentation and noise injection to protect privacy, especially in geolocation and imagery data.
• Used the Cohen’s kappa statistic for evaluation in imbalanced regions to ensure fair performance assessment across diverse socioeconomic contexts.

Experimental results

Research questions

• RQ1 How can machine learning models be consistently evaluated across diverse, real-world SDG monitoring tasks with varying data modalities and sparsity?
• RQ2 To what extent can meta-learning improve model generalization in low-resource regions like Sub-Saharan Africa for land cover classification?
• RQ3 Can self-supervised and multi-task learning methods improve performance on SDG-related tasks with limited labeled data?
• RQ4 How do biases in model predictions—such as overestimating wealth in poor regions—affect policy-relevant applications of ML in development?
• RQ5 What are the privacy risks in releasing geospatial and survey data for SDG monitoring, and how can they be mitigated?

Key findings

• The MAML-trained model achieved a test accuracy of 0.74, F1-score of 0.72, and Cohen’s kappa of 0.32 on land cover classification across Sub-Saharan Africa, demonstrating the potential of meta-learning in low-resource settings.
• 11 out of 15 datasets in SustainBench are released publicly for the first time, significantly increasing data accessibility for SDG research.
• The use of the kappa statistic instead of accuracy or F1-score provides a more robust evaluation metric in regions with highly imbalanced class distributions.
• Baseline models on poverty prediction and crop yield estimation show strong performance, indicating that ML can effectively leverage satellite and mobile data for SDG indicators.
• Privacy-preserving techniques—such as geolocation jittering, image blurring, and data aggregation—were successfully applied to protect individual identities in survey and imagery data.
• The benchmark supports advanced methodological research, including self-supervised learning and multi-modal learning, by providing diverse, real-world datasets with standardized evaluation protocols.