[Paper Review] Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model

Danfeng Hong, Jingliang Hu+3 more|arXiv (Cornell University)|May 21, 2021

Remote-Sensing Image Classification55 references4 citations

TL;DR

This paper proposes a Shared and Specific Feature Learning (S2FL) model for multimodal remote sensing data, decomposing inputs into modality-shared and modality-specific features to improve land cover classification. Evaluated on three new benchmark datasets—Houston2013 (HS-MS), Berlin (HS-SAR), and Augsburg (HS-SAR-DSM)—S2FL achieves state-of-the-art performance, with 83.36% overall accuracy on the three-modal dataset, outperforming existing methods.

ABSTRACT

As remote sensing (RS) data obtained from different sensors become available largely and openly, multimodal data processing and analysis techniques have been garnering increasing interest in the RS and geoscience community. However, due to the gap between different modalities in terms of imaging sensors, resolutions, and contents, embedding their complementary information into a consistent, compact, accurate, and discriminative representation, to a great extent, remains challenging. To this end, we propose a shared and specific feature learning (S2FL) model. S2FL is capable of decomposing multimodal RS data into modality-shared and modality-specific components, enabling the information blending of multi-modalities more effectively, particularly for heterogeneous data sources. Moreover, to better assess multimodal baselines and the newly-proposed S2FL model, three multimodal RS benchmark datasets, i.e., Houston2013 -- hyperspectral and multispectral data, Berlin -- hyperspectral and synthetic aperture radar (SAR) data, Augsburg -- hyperspectral, SAR, and digital surface model (DSM) data, are released and used for land cover classification. Extensive experiments conducted on the three datasets demonstrate the superiority and advancement of our S2FL model in the task of land cover classification in comparison with previously-proposed state-of-the-art baselines. Furthermore, the baseline codes and datasets used in this paper will be made available freely at https://github.com/danfenghong/ISPRS_S2FL.

Motivation & Objective

• To address the challenge of fusing heterogeneous multimodal remote sensing data with varying resolutions, sensors, and imaging characteristics for improved land cover classification.
• To overcome the limitations of existing feature learning methods that concatenate features without effective fusion or modality-specific representation.
• To provide standardized, publicly available benchmark datasets to accelerate research in multimodal remote sensing and feature learning.
• To develop a novel feature learning model that explicitly separates shared and specific components across modalities for enhanced representation learning.

Proposed method

• Proposes a Shared and Specific Feature Learning (S2FL) model that decomposes multimodal remote sensing data into shared features (common to all modalities) and specific features (unique to each modality).
• Uses a joint optimization framework to learn shared and specific features simultaneously, preserving modality-specific information while enabling cross-modal alignment.
• Applies sparsity-promoting ℓ₁-norm regularization to enhance feature selection and improve robustness in multimodal feature learning.
• Employs a co-space learning strategy with shared and specific components, enabling effective fusion of heterogeneous data such as hyperspectral, multispectral, SAR, and DSM.
• Integrates a discriminative classifier on the fused features to perform land cover classification.
• Releases three new benchmark datasets: Houston2013 (HS-MS), Berlin (HS-SAR), and Augsburg (HS-SAR-DSM), with standardized preprocessing and evaluation protocols.

Experimental results

Research questions

• RQ1 How can multimodal remote sensing data with heterogeneous characteristics (e.g., different resolutions, sensors, and imaging principles) be effectively fused to improve land cover classification?
• RQ2 To what extent does separating shared and specific features across modalities enhance feature representation and classification accuracy compared to conventional concatenation-based methods?
• RQ3 Can a unified feature learning model generalize across different combinations of modalities (e.g., two or three modalities) while maintaining high performance and robustness?
• RQ4 How do sparsity-promoting regularization and shared-specific decomposition influence the model’s sensitivity to training set size and generalization capability?

Key findings

• The S2FL model achieves 83.36% overall accuracy on the HS-SAR-DSM (Augsburg) dataset, significantly outperforming the next-best method, ℓ₁-CoSpace, which achieves 81.49%.
• On the HS-SAR-DSM dataset, S2FL achieves 61.38% average accuracy and 0.7626 kappa coefficient, demonstrating superior performance over both ℓ₂-CoSpace and ℓ₁-CoSpace.
• The ablation study confirms that S2FL consistently outperforms ℓ₂-CoSpace and ℓ₁-CoSpace across all modality combinations, including two- and three-modal settings.
• The use of ℓ₁-norm regularization in feature learning leads to observable improvements over ℓ₂-norm, indicating that sparsity promotes better fusion and representation.
• The model shows enhanced robustness and interpretability compared to deep learning baselines, particularly in low-data regimes and with increasing modality heterogeneity.
• The release of three standardized benchmark datasets (Houston2013, Berlin, Augsburg) provides a foundation for reproducible and comparable evaluation of future multimodal remote sensing methods.