[Paper Review] High-Resolution Building and Road Detection from Sentinel-2

Wojciech Sirko, Emmanuel Asiedu Brempong+10 more|arXiv (Cornell University)|Oct 17, 2023

Automated Road and Building Extraction14 citations

TL;DR

The paper trains a student model to predict 50 cm building/road segmentation from stacks of 10 m Sentinel-2 images by imitating a high‑resolution teacher model, achieving 78.3% mIoU for buildings (vs 85.3% for the teacher) and building counts with R2=0.91.

ABSTRACT

Mapping buildings and roads automatically with remote sensing typically requires high-resolution imagery, which is expensive to obtain and often sparsely available. In this work we demonstrate how multiple 10 m resolution Sentinel-2 images can be used to generate 50 cm resolution building and road segmentation masks. This is done by training a `student' model with access to Sentinel-2 images to reproduce the predictions of a `teacher' model which has access to corresponding high-resolution imagery. While the predictions do not have all the fine detail of the teacher model, we find that we are able to retain much of the performance: for building segmentation we achieve 79.0\% mIoU, compared to the high-resolution teacher model accuracy of 85.5\% mIoU. We also describe two related methods that work on Sentinel-2 imagery: one for counting individual buildings which achieves $R^2 = 0.91$ against true counts and one for predicting building height with 1.5 meter mean absolute error. This work opens up new possibilities for using freely available Sentinel-2 imagery for a range of tasks that previously could only be done with high-resolution satellite imagery.

Motivation & Objective

• Motivate accessible, large-scale mapping of buildings and roads without relying on costly high-resolution imagery.
• Develop an end-to-end framework that learns to reproduce high-resolution predictions from low-resolution Sentinel-2 stacks.
• Leverage a teacher model trained on 50 cm imagery to supervise a student model using Sentinel-2 inputs.
• Enable counting of buildings in a patch via centroid prediction to approximate counts.
• Contribute to Open Buildings dataset development by expanding Sentinel-2 based analysis capabilities.

Proposed method

• Use a teacher–student setup where the teacher operates on 50 cm imagery and the student receives a stack of Sentinel-2 frames at 10 m resolution to predict high-resolution semantic masks.
• Employ an encoder–decoder architecture based on HRNet; adapt the first block to preserve higher spatial resolution for LR inputs.
• Fuse temporal information across 32 Sentinel-2 frames via cross-time depthwise convolutions in a residual configuration to capture temporal cues.
• Train a multi-task model outputting building segmentation, road segmentation, building centroids (for counts), and a super-resolved grayscale image to aid registration.
• Use per-pixel Kullback–Leibler Divergence loss with an alignment step that registers label to model output via translation search; include an upsampling-based decoder to reach 50 cm targets.
• Incentivize accurate counts through a centroid-based counting approach, deriving tile counts by summing centroid channel outputs and scaling.

Figure 1: Example operation of our model, where multiple frames of low-resolution Sentinel-2 imagery are used to make a single frame of high-resolution predictions for a variety of output types. A high-resolution image of the same scene is shown for comparison.

Experimental results

Research questions

• RQ1 Can Sentinel-2 stacks at 10 m resolution be used to predict semantic 50 cm building and road masks?
• RQ2 How close can Sentinel-2 based predictions approach high-resolution teacher models in mIoU and spatial detail?
• RQ3 Is building counting feasible from Sentinel-2 predictions, and how does it compare to true counts?
• RQ4 What is the impact of temporal framing and pairing strategies on downstream segmentation performance?
• RQ5 How does input/output/label resolution affect performance and how does Sentinel-2 based counting perform across scales?

Key findings

• Building segmentation achieves 78.3% mIoU with Sentinel-2 supervision vs 85.3% mIoU for the high-resolution teacher.
• The building-count task attains R2 = 0.91 against true counts, closely matching the teacher baseline of R2 = 0.95.
• Using 4 m input resolution yields performance comparable to a single-frame high-resolution model trained on 4 m data, and the best Sentinel-2 based model reaches 50 cm output with notable accuracy.
• Performance improves with more timeframes; 32-frame stacks outperform single-frame setups by about 5 percentage points in mIoU for buildings.
• Pairing each timeframe with the 17th frame (nearest in time to the teacher label) provides a significant boost over no pairing; cross-time fusion further enhances results.
• Training scale matters: increasing the training data from 1% to 100% improves building mIoU from 69.1 to 76.6 (with 32 frames).

Figure 2: Examples of building and road detection from Sentinel-2 imagery, each covering an area of $192^{2}$ m 2 . The panels on the left show high-resolution satellite imagery of the scene for comparison; although Sentinel-2 imagery has much lower level of detail in each frame, we are able to pred