[Paper Review] A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery

Esther Rolf, Jonathan Proctor+6 more|arXiv (Cornell University)|Oct 16, 2020

Remote-Sensing Image ClassificationEngineering36 references107 citations

TL;DR

This paper introduces MOSAIKS, a generalizable and computationally efficient method for satellite image machine learning that decouples image featurization from task-specific modeling. By pre-computing a single, task-agnostic set of random convolutional features on a GPU, researchers can achieve state-of-the-art performance using only linear regression on a personal computer, reducing user-level computation by up to 10,000× compared to deep learning while maintaining competitive accuracy across diverse global tasks like forest cover, poverty, and infrastructure estimation.

ABSTRACT

Combining satellite imagery with machine learning (SIML) has the potential to address global challenges by remotely estimating socioeconomic and environmental conditions in data-poor regions, yet the resource requirements of SIML limit its accessibility and use. We show that a single encoding of satellite imagery can generalize across diverse prediction tasks (e.g. forest cover, house price, road length). Our method achieves accuracy competitive with deep neural networks at orders of magnitude lower computational cost, scales globally, delivers label super-resolution predictions, and facilitates characterizations of uncertainty. Since image encodings are shared across tasks, they can be centrally computed and distributed to unlimited researchers, who need only fit a linear regression to their own ground truth data in order to achieve state-of-the-art SIML performance.

Motivation & Objective

• Address the high computational and technical barriers limiting access to satellite image machine learning (SIML) in low-income and data-poor regions.
• Overcome the inefficiency of reprocessing satellite imagery for each new task by centralizing the computationally intensive featurization step.
• Enable non-experts to achieve state-of-the-art SIML performance using only basic computing resources and their own labeled data.
• Demonstrate that a single, unsupervised image encoding can generalize across diverse socioeconomic and environmental prediction tasks globally.
• Reduce the overall computational burden of the SIML ecosystem by caching and reusing shared features across users and tasks.

Proposed method

• Apply random convolutional features (RCF) to transform high-resolution daytime satellite images into fixed, K-dimensional feature vectors in an unsupervised manner.
• Perform the computationally intensive featurization step once on a centralized GPU, generating a shared feature matrix X for all users.
• Store the pre-computed features in tabular form, enabling distribution to researchers without requiring image access or processing.
• Allow individual users to merge their own task-specific ground truth labels (y) with the shared features (X) and train a simple linear regression model (e.g., ridge regression) on a personal computer.
• Use a fixed, non-trainable RCF encoder to ensure consistency and avoid task-specific tuning or fine-tuning of deep neural networks.
• Leverage the fact that satellite images are collected from a fixed distance and viewing angle, enabling simpler, more efficient feature extraction than methods designed for natural images.

Experimental results

Research questions

• RQ1 Can a single, unsupervised image encoding generalize across diverse satellite image prediction tasks such as forest cover, house prices, and road length?
• RQ2 To what extent can a linear model trained on pre-computed features match the performance of complex deep neural networks in SIML applications?
• RQ3 How much computational savings can be achieved by decoupling image featurization from task-specific model training in SIML?
• RQ4 Can this approach significantly lower the barrier to entry for researchers and practitioners without access to high-performance computing or remote sensing expertise?
• RQ5 How does the performance of the shared feature representation compare to transfer learning and other state-of-the-art methods across multiple global datasets?

Key findings

• MOSAIKS achieves state-of-the-art performance across diverse tasks—including forest cover, house prices, and population density—using only linear regression on a personal computer.
• The method reduces user-level computation time by up to 10,000× compared to fine-tuned ResNet models, with model training taking only ~50 seconds on a 10-core CPU or ~2.8 seconds on a GPU.
• For a single task, the total ecosystem cost of MOSAIKS is 5.3× faster than ResNet, and the efficiency gain increases with the number of tasks due to feature reuse.
• The pre-computed features enable label super-resolution, allowing predictions at higher spatial resolution than the original labels.
• In cross-validation, MOSAIKS outperforms or matches transfer learning baselines on tasks like child malnutrition, wealth, and electricity access in Rwanda, Haiti, and Nepal, with R² values reaching up to 0.8 across multiple metrics.
• The approach enables uncertainty characterization and diagnostic analyses, such as feature importance and model interpretability, without requiring reprocessing of imagery.