[Paper Review] A Spacecraft Dataset for Detection, Segmentation and Parts Recognition

Dung Anh Hoang, Bo Chen+1 more|arXiv (Cornell University)|Jun 15, 2021

Advanced Neural Network Applications38 references6 citations

TL;DR

This paper introduces the first publicly available dataset for spacecraft detection, instance segmentation, and part recognition, comprising 3,117 annotated images of satellites and space stations with bounding boxes, instance masks, and fine-grained part-level annotations. The authors employ a bootstrap annotation strategy combining automated labeling and iterative human refinement, and benchmark state-of-the-art models, showing that existing models trained on Earth-based datasets underperform significantly on space objects, especially for complex parts like antennas.

ABSTRACT

Virtually all aspects of modern life depend on space technology. Thanks to the great advancement of computer vision in general and deep learning-based techniques in particular, over the decades, the world witnessed the growing use of deep learning in solving problems for space applications, such as self-driving robot, tracers, insect-like robot on cosmos and health monitoring of spacecraft. These are just some prominent examples that has advanced space industry with the help of deep learning. However, the success of deep learning models requires a lot of training data in order to have decent performance, while on the other hand, there are very limited amount of publicly available space datasets for the training of deep learning models. Currently, there is no public datasets for space-based object detection or instance segmentation, partly because manually annotating object segmentation masks is very time consuming as they require pixel-level labelling, not to mention the challenge of obtaining images from space. In this paper, we aim to fill this gap by releasing a dataset for spacecraft detection, instance segmentation and part recognition. The main contribution of this work is the development of the dataset using images of space stations and satellites, with rich annotations including bounding boxes of spacecrafts and masks to the level of object parts, which are obtained with a mixture of automatic processes and manual efforts. We also provide evaluations with state-of-the-art methods in object detection and instance segmentation as a benchmark for the dataset. The link for downloading the proposed dataset can be found on https://github.com/Yurushia1998/SatelliteDataset.

Motivation & Objective

• To address the critical lack of publicly available, high-quality space-based datasets for deep learning in aerospace applications.
• To enable research in spacecraft detection, instance segmentation, and fine-grained part recognition using real and synthetic space imagery.
• To reduce annotation cost through a bootstrap strategy combining automated labeling and iterative human refinement.
• To establish a benchmark for evaluating state-of-the-art models on space-specific visual challenges.

Proposed method

• The dataset is constructed from 3,117 real and synthetic images of satellites and space stations collected from various sources.
• A hybrid annotation pipeline is used: initial manual labeling with interactive tools, followed by model-based coarse labeling and subsequent human refinement.
• A bootstrap strategy iteratively improves model predictions on new images using previously labeled data, scaling up annotation efficiency.
• State-of-the-art deep learning models—including DeepLabV3+, OCRNet, HRNet, and ResNeSt—are trained and evaluated on the dataset using standard training protocols.
• Images are resized to 1024×512 for training and 2048×1024 for validation, with standard optimization (SGD, weight decay) and auxiliary heads for improved performance.
• Performance is evaluated using mIoU across all classes and per-part categories, with separate metrics including and excluding the background class.

Experimental results

Research questions

• RQ1 How does the performance of state-of-the-art segmentation models on spacecraft images compare to their performance on standard Earth-based datasets like Cityscapes and Pascal VOC?
• RQ2 To what extent can automated and semi-automated annotation techniques reduce the cost and time of creating pixel-level segmentation masks for space objects?
• RQ3 What are the key challenges in segmenting spacecraft parts such as antennas, solar panels, and main bodies due to their complex and variable shapes?
• RQ4 Can models pre-trained on Earth-based data generalize effectively to space-based visual tasks without fine-tuning or domain adaptation?
• RQ5 How does the inclusion of background pixels affect the evaluation of segmentation models on space imagery, and what is the performance gap when excluding it?

Key findings

• The proposed dataset achieves a mean intersection-over-union (mIoU) of 0.790 for the ResNeSt200 model on the full spacecraft segmentation task, with 0.767 for the main body, 0.878 for solar panels, and only 0.640 for antennas.
• The mIoU for spacecraft part segmentation is significantly lower for antennas (0.585–0.650) compared to the body (0.834–0.842) and solar panels (0.839–0.878), indicating high difficulty in identifying complex, irregularly shaped parts.
• State-of-the-art models trained on Earth-based datasets like Cityscapes and Pascal VOC achieve average mIoU scores of 0.859 and 0.824, respectively, which are 6% to 11% higher than the best-performing model on the proposed spacecraft dataset.
• The performance gap between Earth-based and space-based segmentation tasks is substantial, with the best model on the new dataset achieving only 74.9% of the average mIoU of models on Pascal VOC.
• Qualitative results show that all models struggle to accurately predict antenna masks, especially in complex configurations, confirming the quantitative mIoU results and highlighting the need for domain-specific model design.
• The bootstrap annotation strategy successfully reduces manual effort while maintaining high annotation quality, enabling the creation of a large-scale, fine-annotated dataset with limited human labor.