[Paper Review] High-Resolution Multispectral Dataset for Semantic Segmentation

Ronald Kemker, Carl Salvaggio+1 more|arXiv (Cornell University)|Mar 6, 2017

Remote-Sensing Image Classification11 references21 citations

TL;DR

This paper introduces a high-resolution (4.7 cm GSD) multispectral dataset acquired via UAS, featuring 18 unbalanced classes and pre-split training, validation, and testing folds to enable standardized benchmarking of semantic segmentation models for non-RGB remote sensing imagery. The dataset presents significant challenges due to high spatial variability, large pixel count, and class imbalance, making it a demanding benchmark for deep learning frameworks aiming for real-time deployment.

ABSTRACT

Unmanned aircraft have decreased the cost required to collect remote sensing imagery, which has enabled researchers to collect high-spatial resolution data from multiple sensor modalities more frequently and easily. The increase in data will push the need for semantic segmentation frameworks that are able to classify non-RGB imagery, but this type of algorithmic development requires an increase in publicly available benchmark datasets with class labels. In this paper, we introduce a high-resolution multispectral dataset with image labels. This new benchmark dataset has been pre-split into training/testing folds in order to standardize evaluation and continue to push state-of-the-art classification frameworks for non-RGB imagery.

Motivation & Objective

• To address the lack of publicly available, high-resolution, labeled multispectral datasets for semantic segmentation in remote sensing.
• To enable standardized evaluation of deep learning models on non-RGB imagery by providing pre-split training, validation, and testing sets.
• To support the development of deployable, real-time semantic segmentation frameworks for unmanned aircraft systems (UAS) imagery.
• To provide a benchmark that reflects real-world challenges such as class imbalance, spatial variability, and small/hard-to-detect objects.
• To facilitate the transfer of pre-trained models from RGB-based datasets (e.g., ImageNet) to multispectral remote sensing applications.

Proposed method

• The dataset was collected using a Tetracam Micro-MCA6 multispectral sensor mounted on a UAS over Hamlin Beach State Park, achieving a ground sample distance (GSD) of 4.7 cm.
• The dataset contains 18 unbalanced semantic classes and is pre-split into training, validation, and testing sets to prevent data leakage and ensure generalization evaluation.
• A multi-stage classification pipeline was evaluated, including SVM with various band combinations (RGB, NIR, CIR, VNIR, and full 6-band), and a stacked autoencoder (SCAE) for feature extraction.
• Feature extraction used a stacked autoencoder with WPCA (weighted principal component analysis) to reduce dimensionality while preserving 99% of variance.
• Target detection was evaluated using standard algorithms: spectral angle mapper (SAM), spectral matched filter (SMF), constrained-energy minimization (CEM), and adaptive cosine estimator (ACE).
• Performance was measured using overall accuracy (OA), mean class accuracy (AA), kappa statistic (κ), and area under the curve (AUC) for target detection.

Experimental results

Research questions

• RQ1 Can a high-resolution multispectral dataset with pre-split training and testing folds enable reliable and standardized benchmarking of semantic segmentation models for non-RGB remote sensing?
• RQ2 How does the inclusion of near-infrared (NIR) bands affect classification performance on a multispectral dataset with high vegetation content?
• RQ3 To what extent do traditional machine learning models (e.g., SVM with hand-selected features) perform on high-resolution, unbalanced multispectral data compared to deep learning approaches?
• RQ4 Can end-to-end deep learning frameworks trained on this dataset achieve real-time inference performance suitable for UAS deployment?
• RQ5 How do standard target detection algorithms (SAM, SMF, CEM, ACE) perform on white and black wooden panels under varying illumination conditions in multispectral data?

Key findings

• The SVM using all six multispectral bands achieved the highest overall accuracy (61.3%) and kappa statistic (0.538), outperforming RGB-only (36.4% OA) and NIR-only (59.6% OA) configurations.
• The inclusion of near-infrared (NIR) bands significantly improved classification performance, with NIR-only SVM achieving 59.6% OA, indicating strong vegetation discrimination capability.
• The SCAE-based feature extraction method outperformed the MICA model, suggesting that low-level features over a larger receptive field are more effective than higher-level features in this context.
• For target detection, the SAM and ACE algorithms achieved the highest AUC scores (0.91–0.98) on both black and white targets, especially on the test set, indicating strong robustness to illumination variation.
• The SMF (two-sided) algorithm showed poor performance on black targets (AUC = 0.613), highlighting the sensitivity of some algorithms to target contrast and background conditions.
• The dataset’s high spatial resolution and large pixel count (over 100 million pixels) impose significant computational demands, requiring GPU acceleration and limiting end-to-end training on full scenes.