[Paper Review] SpaceNet: A Remote Sensing Dataset and Challenge Series

Adam Van Etten, Dave Lindenbaum+1 more|arXiv (Cornell University)|Jul 3, 2018

Automated Road and Building ExtractionEngineering20 references356 citations

TL;DR

SpaceNet releases a large labeled satellite imagery dataset on AWS and a series of public challenges for automated building footprint and road network extraction, with novel metrics for roads (APLS) and high-performing submissions.

ABSTRACT

Foundational mapping remains a challenge in many parts of the world, particularly in dynamic scenarios such as natural disasters when timely updates are critical. Updating maps is currently a highly manual process requiring a large number of human labelers to either create features or rigorously validate automated outputs. We propose that the frequent revisits of earth imaging satellite constellations may accelerate existing efforts to quickly update foundational maps when combined with advanced machine learning techniques. Accordingly, the SpaceNet partners (CosmiQ Works, Radiant Solutions, and NVIDIA), released a large corpus of labeled satellite imagery on Amazon Web Services (AWS) called SpaceNet. The SpaceNet partners also launched a series of public prize competitions to encourage improvement of remote sensing machine learning algorithms. The first two of these competitions focused on automated building footprint extraction, and the most recent challenge focused on road network extraction. In this paper we discuss the SpaceNet imagery, labels, evaluation metrics, prize challenge results to date, and future plans for the SpaceNet challenge series.

Motivation & Objective

• Provide a large, labeled remote sensing dataset to accelerate automated map feature extraction.
• Enable public benchmarking of building footprint and road network extraction algorithms.
• Introduce evaluation metrics that reflect routing utility (APLS) beyond pixel-based metrics.
• Demonstrate improvements from Challenge 1 to Challenge 2 and establish a roadmap for future challenges.

Proposed method

• Release of SpaceNet imagery and validated labels under CC BY-SA 4.0 on AWS.
• Definition of evaluation metrics: IoU-based F1 for buildings and the APLS graph-based metric for roads.
• Ground-truth labeling pipelines with QA/QC to ensure near-5-pixel corner accuracy for buildings and 7-pixel centerline accuracy for roads.
• Ground-truth road labeling aligned with OpenStreetMap guidelines to enable routable networks.

Experimental results

Research questions

• RQ1 How accurately can automated methods extract building footprints from high-resolution satellite imagery?
• RQ2 Can automated road network extraction produce routable graphs suitable for routing applications?
• RQ3 Do novel graph-based metrics (APLS) better capture routing-relevant quality than pixel-based metrics?
• RQ4 How does algorithm performance improve over SpaceNet challenges and across cities with varying building densities and road types?

Key findings

• Challenge 1 baseline F1 ~0.21 for Rio de Janeiro building footprints.
• Challenge 2 improved F1 scores across cities, with top entrants achieving around 0.69 total F1 (Las Vegas 0.89, Paris 0.75, Shanghai 0.60, Khartoum 0.54).
• Challenge 3 road networks yielded an Average Path Length Similarity (APLS) total score around 0.666 for top entrants, indicating effective routing-relevant graph accuracy.
• SpaceNet provides data and challenges that produce progressively better building footprint extraction performance and demonstrate the utility of a graph-based road metric (APLS) for routing.
• The winning road-network score was 0.6663 (albu) across four AOIs.