[Paper Review] Model Generalization in Deep Learning Applications for Land Cover Mapping

Lucas ZhongMing Hu, Caleb Robinson+1 more|arXiv (Cornell University)|Aug 9, 2020

Remote-Sensing Image Classification7 references4 citations

TL;DR

This paper investigates the generalization limits of deep learning models in land cover mapping across different continents and seasons using the SEN12MS dataset. It demonstrates that models trained on one continent or season often fail on others due to landscape distribution shifts, and proposes clustering satellite imagery by spectral bands to identify region-specific visual patterns that explain poor generalization.

ABSTRACT

Recent work has shown that deep learning models can be used to classify land-use data from geospatial satellite imagery. We show that when these deep learning models are trained on data from specific continents/seasons, there is a high degree of variability in model performance on out-of-sample continents/seasons. This suggests that just because a model accurately predicts land-use classes in one continent or season does not mean that the model will accurately predict land-use classes in a different continent or season. We then use clustering techniques on satellite imagery from different continents to visualize the differences in landscapes that make geospatial generalization particularly difficult, and summarize our takeaways for future satellite imagery-related applications.

Motivation & Objective

• To evaluate how well deep learning models trained on one continent or season generalize to other continents or seasons in land cover classification.
• To identify the root causes of poor generalization in geospatial deep learning, particularly due to landscape and sensor variability.
• To develop a method for predicting model performance on new regions using landscape cluster distributions derived from satellite imagery.
• To guide practitioners in selecting appropriate models or data augmentation strategies for out-of-distribution geospatial applications.

Proposed method

• Trained FC-DenseNet models on land cover classification using the SEN12MS dataset, with training and evaluation split by continent and season.
• Applied k-means clustering to image patches based on mean Sentinel-1 and Sentinel-2 band values to identify distinct landscape types.
• Generated per-continent histograms of cluster frequencies to quantify landscape distribution differences across continents.
• Used PCA on normalized cluster histograms to visualize similarity and dissimilarity between continental landscape profiles.
• Calculated conditional probabilities P(continent|cluster) to determine which landscape types are indicative of specific continents.
• Evaluated model generalization performance across continents and seasons to correlate performance drops with landscape distribution shifts.

Experimental results

Research questions

• RQ1 How well do models trained on one season generalize to other seasons?
• RQ2 How well do models trained on one continent generalize to other continents?
• RQ3 What features or indicators can predict when model generalization will be poor?
• RQ4 Can landscape clustering from multispectral imagery explain cross-continental generalization failures?

Key findings

• Out-of-continent and out-of-season model performance is significantly lower than in-continent and in-season performance, indicating poor generalization across regions and times.
• Models trained on Europe generalized reasonably well to North America, and models trained on South America generalized well to Africa, which correlates with similar landscape distributions observed in cluster histograms.
• Australia’s landscapes are visually distinct from all other continents, and models trained on other continents performed poorly on Australian data, indicating a unique landscape distribution.
• Certain landscape clusters—such as desert (Clusters 5, 10) and snow/tundra (Clusters 2, 9)—are highly indicative of specific continents, with P(continent|cluster) showing strong localization.
• The PCA visualization of normalized cluster histograms confirmed that Africa and South America are similar in landscape composition, as are Europe and North America, while Asia lies in a transitional position.
• Models trained on a single continent or season often fail on new regions due to unaccounted landscape variability, suggesting that data diversity in training is critical for robustness.